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Methods and Models for 
Life Cycle Costing 
(RTO-TR-SAS-054) 


Executive Summary 


The core objectives of the SAS-054 Task Group were to understand NATO and PfP Nations’ methods and 
models and to promulgate best practice in life cycle costing that can be adopted throughout the various 
NATO Phased Armaments Programming System (PAPS) phases. This is particularly important where 
decisions are made when the amount of supporting information to provide robust life cycle costs are 
sparse. 


There are many methods and models available to conduct life cycle cost analysis and it was important to 
understand the applicability and boundaries of each method and model in order to use them appropriately. 


The report contains a review of cost forecasting methods and models together with examples to provide a 
comprehensive guideline in the application of life cycle costing and particularly in its use within multi- 
national programmes. 


It is accepted that a life cycle cost estimate of weapon systems is inherently uncertain and will contain 
risk. The treatment of uncertainty and risk within the context of developing the life cycle cost estimate is 
also explained within the report. 


The report therefore provides the reader with a comprehensive view on the application and use of life 
cycle costing from an early conceptual phase in the product life cycle through to disposal. It provides 
illustrations on the types of life cycle cost studies that can be conducted and examples to demonstrate the 
benefits. The report concludes with a number of recommendations to improve the use and understanding 
of life cycle costing in the decision making process. 
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Méthodes et modeéles d’évaluation 
du coiit de possession 
(RTO-TR-SAS-054) 


Synthese 


Les objectifs principaux du Groupe de Travail SAS-054 furent de comprendre les méthodes et modéles de 
VOTAN et des nations du Partenariat pour la Paix (PfP), et de faire valoir les bonnes pratiques en matiére 
d’évaluation du cotit de possession, qui pourraient étre adoptées dans les différentes phases PAPS 
(systeme de programmation échelonnée des armements) de l’OTAN. Ceci est particuliérement important 
quand des décisions sont prises lorsque la somme des informations permettant d’évaluer le cofit de 
possession est faible. 


Il existe de nombreuses méthodes et modéles disponibles pour ce faire ; et il est important de comprendre 
Vapplicabilité et les limites de chaque méthode ou modéle pour les utiliser 4 bon escient. 


Ce rapport examine des méthodes et modéles de prédiction de cofits, ainsi que des exemples servant de 
guide pour l’évaluation du cofit de possession, et plus particuli¢rement dans le cadre d’utilisations 
multinationales. 


Il est généralement admis que l’estimation du cofit de possession d’un systeme d’arme est de fait 
incertaine et qu’elle comporte des risques. Le traitement de cette incertitude et de ce risque dans le 
contexte du développement de |’estimation du coat de possession est aussi expliqué dans ce rapport. 


Ce rapport fournit donc au lecteur une vue complete sur l’application et utilisation de l’analyse du colt 
de possession depuis une phase conceptuelle avancée dans le cycle de vie d’un systéme d’arme, jusqu’a 
son retrait. Il illustre les études sur les analyses des cofits de possession qui pourraient étre menées et des 
exemples démontrant ses avantages. Ce rapport conclut par un certain nombre de recommandations pour 
améliorer |’ utilisation et la compréhension de I’ analyse du cofit de possession dans la prise de décision. 
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OVERVIEW 


The results of life cycle costing must, whatever the phase of the programme, contribute to the process by 
which managers can make the best decisions on options presented to them. These options can include 
evaluation of future expenditure, comparison between alternative solutions, management of existing 
budgets, options for procurement and evaluation of cost reduction opportunities. Life cycle costing is also 
used for affordability assessment and determining the cost drivers associated with the Key Performance 
Indicators or Key User Requirements. There are many methods and models available to conduct life cycle 
cost estimates. It is important to understand the applicability and boundaries of each method and model in 
order to use them appropriately. 


The core objectives of the SAS-054 Task Group were to understand NATO and PfP Nations’ methods and 
models and to promulgate best practice within the NATO Phased Armaments Programming System 
(PAPS) phases. In order to meet this aspiration the following objectives were defined in the Task Group 
terms of reference. 


O.1 REVIEW OF COST FORECASTING METHODS 


The report has captured all the key estimating methods and provided examples to demonstrate their 
applicability. For consistency, both the methods and models have been categorised as Optimisation, 
Simulation, Estimation and Decision Support. The findings confirmed that almost all nations used a 
similar process to develop life cycle cost estimates; that the quality of the available data nearly always 
determined the method to be employed; and, in addition, that the type of study also influenced the process 
and the selection of the appropriate method. 


0.2 REVIEW OF COST FORECASTING MODELS 


In developing life cycle cost estimates all the nations have in-house developed models that are based on a 
defined Cost Breakdown Structure. Data for these models is estimated either by empiric methods or 
parametric formulae (for completeness, sometimes both techniques are employed). The findings confirmed 
that there were many life cycle cost models in use and these are identified within the report. Generally 
speaking, the use of more than one model to produce a life cycle cost estimate is considered good practice. 
This would provide verification of the life cycle cost estimate. However, the use of multiple methods and 
models should always be balanced with the knowledge and understanding of how the estimate will be 
used. It is important to ensure that the life cycle costing activities are conducted in a cost-effective manner 
and balanced with what is realistically achievable at a specific stage in the programme. It is also important 
to ensure that every model used for acquisition and life cycle costing is subject to calibration, verification 
and validation. This will build confidence that the cost model is fit for purpose. 


0.3. GUIDELINES FOR THE COLLECTION AND UNDERSTANDING OF 
COST RELATED DATA FOR NATIONAL AND MULTI-NATIONAL 
PROGRAMMES 


In terms of time, effort, and resources consumed, collection of data is a major part of a life cycle cost 
study. Life cycle costing is a data driven process, as the amount, quality and other characteristics of the 
available data often define what methods and models can be applied, what analyses can be performed, 
and therefore determine the usefulness of the results that can be achieved. Unfortunately, because 
uncertainty, risks, and opportunities decrease as the life cycle progresses, the need for knowledge is 
greatest at the earliest stages. This means that more time and resources should be allocated to the data 
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collection effort during the earlier stages of the life cycle in order to develop an acceptable and auditable 
life cycle cost estimate. 


0.4 TREATMENT OF UNCERTAINTY AND RISK 


Life cycle cost estimates of defence programmes are inherently uncertain and risky. Estimates are often 
made when information and data is sparse. Estimates, in turn, are based on historical samples of data that 
are almost always messy, of limited size, and difficult and costly to obtain. And no matter what estimation 
tool or method is used, historical observations never perfectly fit a smooth line or surface, but instead fall 
above and below an estimated value. To complicate matters, the weapon system under study is often of 
sketchy design. 


For all of these reasons, a life cycle cost estimate, when expressed as a single number, is merely one 
outcome or observation in a probability distribution of costs. To better support the decision making 
process it is recommended that three point estimating is always undertaken. This study has captured a 
wide variety of methods and models available for conducting risk and uncertainty analysis of life cycle 
cost estimates of weapons systems. Each, if used properly, can give scientifically sound results and 
provide a better yardstick for an accurate life cycle cost estimate. 


0.5 GUIDELINE FOR MULTI-NATIONAL PROGRAMMES 


Multi-national programmes involve at least two nations who have agreed upon the main principles of 
co-operation in a Memorandum of Understanding, or an equivalent arrangement, for one or several phases 
of the entire lifetime of that programme. It has been found that the life cycle cost studies for multi-national 
programmes follow the same principles as national life cycle cost requirements. However, there are some 
specifics that have to be taken into account in terms of organisation, currency issues, studies, model(s) and 
presentation of the life cycle cost results. 


0.6 REVIEW AND POSSIBLE ENHANCEMENT OF THE NATO GENERIC 
COST BREAKDOWN STRUCTURE 


The NATO generic cost breakdown structure developed by SAS-028 has been reviewed by participating 
nations and organisations. Within the report it is recommended that some changes to the NATO generic 
cost breakdown structure be made based on recent OCCAR experience in implementing it on multi- 
national programmes. 


O.7 AWARENESS OF NEW DEVELOPMENTS IN METHODS AND MODELS 


A number of new developments have been identified that will impact on life cycle costing and analysis 
activities. Some appear to stem from changes in the types of studies being conducted (e.g. capability 
estimating, system of systems estimating, etc.). This will not change the way life cycle costing is 
conducted, but it has changed the interpretation of the costing boundary of the estimates. 


In addition, to more effectively manage scarce defence resources, several NATO and PfP nations are 
initiating efforts to analyse the costs, capabilities, and risks within an entire portfolio of assets in a joint 
war-fighting environment. Viewing capabilities across the entire portfolio of assets enables the decision 
makers to make better informed choices about how to reallocate resources with the ultimate goal of 
delivering needed capabilities to the joint force more rapidly and efficiently. Capability portfolios are 
intended to serve as a basis for strategic level trade studies by senior decision makers. 
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It is recommended that further studies be conducted to better understand the life cycle costing 
requirements and benefits to the decision making process that these new developments may bring. 


0.8 CONCLUSION 


The report provides details on the findings of the Task Group into the methods and models being used for 
life cycle costing. In addition, it has examined data collection, the measurement of risk and uncertainty, 
new developments and other related life cycle cost issues. The report concludes with a number of 
recommendations to improve the use and understanding of life cycle costing in the decision making 
process. The most important recommendation is to implement this work as NATO policy. 


0.9 KEY RECOMMENDATIONS 


The following key recommendations are made with regard to the development and improvement in life 
cycle costing for multi-national programmes. 


0.9.1 Life Cycle Costing Methods 


1) Life cycle cost estimates should be fully documented (Sub-section 2.3.2) 


¢ Acost analyst should be able to re-create the complete estimate working from the documentation 
alone. 


e All assumptions and data related to the study should be captured in an MDAL or CARD or 
similar document. 


e Assumptions recorded in an assumptions list such as the MDAL or CARD should be questioned 
by an independent technical team. 


2) All life cycle cost estimates should be prepared by suitably experienced personnel (Sub-section 
5.2.3.1) 


¢ Decisions such as budget setting and options analysis studies are often conducted when data to 
support cost forecasting and life cycle costing is very sparse. It is therefore essential that 
experienced personnel are used to conduct the life cycle cost estimates to support the decision 
process at these key stages. 


3) The life cycle cost analysis should include an affordability analysis (Sub-section 2.9) 


e Affordability plays an important part in programme decisions throughout the life cycle. Even 
before a programme is formally approved for initiation, affordability plays a key role in the 
identification of capability needs. This aspect is part of the process which balances cost versus 
performance and in establishing key performance parameters. Although this is not common 
practice in all nations the assessment of affordability is one that we recommend should be 
conducted by all nations. 


4) Life cycle cost estimates, where possible, should use two independent methods for each cost 
breakdown structure element (Sub-section 4.4) 


¢ The use of two independent methods to develop the life cycle cost estimates will improve the 
confidence in the results and help to validate the outputs. It is accepted that this may be tempered 
by the constraints imposed by a financial threshold (see Sub-section 2.6) or by a simple 
consideration of what the estimate will be used for (e.g. rough cost for initial views or detailed 
costs for decision making). 
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0.9.2 Life Cycle Cost Models 


5) All life cycle cost models should be validated (Sub-section 5.5) 


It is essential that all life cycle cost models implemented through spreadsheets or more advanced 
programming techniques be validated by using recognised testing processes. This will increase 
confidence that the model is fit for purpose and that the input data and results can be assessed 
through a clear audit trail and mathematical reasoning of any cost estimating relationships. 


0.9.3 Data for Life Cycle Costing 


6) 


0.9.4 


7) 


0.9.5 


8) 


Investments should be made to increase the accuracy, visibility, and availability of cost, 
programmatic, technical, and performance data within the NATO/PfP cost analysis community 
(Chapter 6). 


Data collection forms a large part of the life cycle costing activity and significant effort is 
expended to gather and analyse the data so that it is suitable for use in life cycle cost analysis 
studies. Improvements in data exchange standards or even the development of a NATO costing 
database would: 

¢ Improve the quality of the life cycle cost estimate; 

¢ Reduce the effort needed to conduct the life cycle cost estimate; and 

¢ Reduce the time schedule to conduct the life cycle cost estimate. 


Multi-National Programmes 


For multi-national programmes the participating nations should agree on a common LCC 
framework (Sub-section 2.10.6) 


The life cycle cost studies for multi-national programmes follow the same principles as those 
required by a national study. However, there are some specifics that must be taken into account in 
terms of organisation, models and the presentation of results. It is essential that all parties in a 
multi-national programme agree on a common life cycle cost framework. This framework is 
determined by the costing boundary and the tools that will be employed to populate the 
framework. A common framework will provide consistency, comprehensiveness, traceability and 
audit. All are essential to achieve life cycle cost estimates in a timely and responsive manner. 


NATO Generic Cost Breakdown Structure 


Enhancements to the GCBS (generic cost breakdown structure) to improve its use (Chapter 10) 


It has been found that most nations have not adopted the generic cost breakdown structure 
reported in RTO-TR-058 as their national life cycle cost breakdown structure. However, 
the NATO generic cost breakdown structure has been applied on specific multi-national 
programmes and some areas of enhancement are recommended. 


The current structure does not allow the identification of the life cycle cost results over the time 
phasing for national financial and programme contributions. Therefore, it is recommended 
to include two dimensions in addition to the Activity, Product and Resource dimensions. 
These additional dimensions are: 


e Time phasing; and 
¢ National contribution. 


As the coding of the Generic Cost Breakdown is complex for non-experts, it is recommended to 
adopt a Generic Hierarchy for the GCBS. 
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0.9.6 Uncertainty and Risk 


9) Risk and uncertainty analysis should be conducted at the same time as the life cycle cost 
estimate (Sub-section 7.9) 


e Life cycle cost estimates of weapon system acquisition programmes are inherently uncertain and 
risky. To better support senior leadership, some sense of risk and uncertainty needs to be 
presented at the same time as developing the point estimate. This will present the decision maker 
with a comprehensive true view of the programme’s likely eventual outcome. 


10) The results of a life cycle cost estimate should be shown as a three point range of estimates 
(Sub-section 7.9) 


e A life cycle cost estimate is not a single number, but rather a continuum or distribution of possible 
values. 


0.9.7. Further Studies 


The following paragraphs outline recommendations for further studies that would benefit the 
understanding and use of life cycle costing in NATO and multi-national environments. 


e The next logical step would be to demonstrate the proof of concept (methods and models) 
described in the report by using a practical application of the guideline. 


e A typical example could be an existing NATO programme (but only using data that was 
available at the time) and/or any other multi-national programme (e.g. AWACS, AGS, JSF, 
NH-90, FREMM). 


e Further research should be conducted in the area of capability portfolio analysis (see Chapter 9). 
This topic of joint NATO/PfP operational activities is becoming more important to NATO and, 
at present, there is insufficient information on how to evaluate the situation where a number of 
discrete assets share the information/data to provide a total capability solution. 


e An investigation into new methods and databases would support this requirement. 
e Research into the life cycle costs of software. This report has not addressed software cost 
estimating as it was felt that this was a subject in its own right. Many academic studies are being 


conducted into open system architecture, modular construction and system behaviours that 
employ software intensive configurations. 


¢ Much is known about modern techniques in software development, but the issue of assessing 
software quality, reliability and support costs is still vague. 
¢ Life cycle cost estimates are produced for a variety of reasons. It would benefit the NATO 
community to investigate how the cost estimates are being used in the decision making process. 
¢ This could avoid the situation where enormous effort may be spent in generating cost 
estimates when the answer could have been given in a more simplistic and effective manner. 
e Estimating accuracy has been an issue for many years. An evaluation could be conducted that 


studied the delta between the original cost estimates and the actual costs. 


¢ This would provide a benefit by having a definitive document that could provide a view of 
estimating accuracy across a number of procurement processes. 
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Research should be conducted continuously to enhance methods and models for life cycle costing 
(Sub-section 4.4). 


¢ Periodically, the US Department of Defense undertake an initiative to review the basis and 
techniques employed in cost estimating. This is supported by industry, a number of academic 
groups and learned societies. However, these initiatives purely examine techniques that will 
be employed within the US. It would be beneficial to conduct a similar continual review 
across NATO and PfP nations. 


The SAS-054 study gathered information on each nation’s approach and use of models to generate 
life cycle costs. The study did not get a clear comprehension on the range of the functionality that 
could be provided by some of these models. 


e It would be of benefit to look in more detail on how these life cycle costing models generate 
cost for Research and Development, Production, Operating and Support. 


The issue of calibration, verification and validation of cost estimating models is of paramount 
importance. However, little or limited space is given in handbooks on the requirements and 
methods of validating cost models. 


e A study could be initiated to develop a common methodology for validating cost models, 
this would help to ensure cost estimating consistency across NATO and PfP nations on each 
nation’s approach and use of models to generate life cycle costs. 


All life cycle cost estimates are only as good as the data that underpins the estimate. 
Much investment has been made in adopting ERP-systems to support financial and project 
reporting. The use of these systems in providing good quality data to support life cycle cost 
estimating is not clearly known. 


e A study should be conducted to evaluate the benefits or otherwise in adopting an ERP-system 
versus the investment in a bespoke system (e.g. VAMOSC) to assist the life cycle cost data 
collection process and improve cost estimating methods and accuracy. 
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Chapter 1 - INTRODUCTION 


1.1 BACKGROUND 


The Research Task Group on cost structures and life cycle costs for military systems (RTG SAS-028) 
developed a Generic Life Cycle Cost Breakdown Structure (GCBS) and associated definitions that could be 
used in any military programme to construct its own cost breakdown structure (Reference RTO-TR-058). 
The group also conducted an analysis on the way to use life cycle costs in the decision making process. 


The participating nations proposed a follow on activity to examine methods and models for life cycle 
costing. This was assumed to be a set of techniques for cost modelling, cost prediction and analysing the 
life cycle costs of a system at any stage of its life. 


At the November 2002 meeting in Brussels, the SAS Panel decided to form the Exploratory Team 
SAS ET-AH on methods and models for life cycle costing (SAS ET-AH). The SAS Panel recommended 
that the first meeting of the new group must be organised not earlier than March 2003 to allow time for 
national distribution and scrutiny of the SAS-058 Technical Report. 


The first meeting of the SAS ET-AH was held over the 8" and 9" July 2003 in Brussels. The SAS ET-AH 
presented their Terms of Reference (TOR) and Technical Activity Proposal (TAP) for approval at the 
SAS Panel business meeting in November 2003. The team recommended a task group as the most 
appropriate way to conduct this study. The SAS Panel accepted this proposal at its November 2003 
meeting, and at its March 2004 meeting the NATO Research and Technology Board approved a new Task 
Group SAS-054/RTG on “Methods and Models for Life Cycle Costing”. The first meeting of SAS-054 
took place on the 25" and 26" May 2004 at NATO Headquarters, Brussels, BELGIUM. 


1.2 JUSTIFICATION 


There is a long and documented history of both cost growth and estimating optimism within military 
programmes. This is particularly the case for multi-national programmes. The NATO ALP-10 —Guidance 
on Integrated Logistics Support for multi-national equipment projects (ILS) dated June 1990 (Reference 
NATO ALP10) states the following: multi-national equipment projects will be required to implement a 
life cycle cost programme. The purpose of this programme is to ensure that the developed system will 
have the lowest possible life cycle cost consistent with performance and schedule requirements. 
To achieve this goal, operation and support cost estimates assist designers and programme managers to 
focus their attention on those design aspects that drive costs. The process of generating realistic cost 
estimates is based on the application of appropriate methods and models. It is essential that future NATO 
programmes have a framework within which to start generating realistic and consistent life cycle cost 
estimates. The first step in this framework was to develop the generic life cycle cost breakdown structure 
under RTG SAS-028. The next step was to define methods and models within this framework which is the 
subject of this report. 


The review and dissemination of the methods and models for life cycle cost estimates will enhance the 
procurement process where life cycle costs was a constituent part of the decision-making process. It was 
expected that the planned framework would: 


e Ensure consistency of the life cycle costing approach within NATO programmes. 
¢ Reduce the effort needed to conduct the life cycle costing analysis. 
¢ Reduce the time schedule to conduct the life cycle costing analysis. 


e Enhance individual nation’s life cycle costing practices. 
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¢ Provide an understanding of NATO and PfP nations’ methods and models. 


¢ Provide guidance to nations not familiar with life cycle costing. 


13 LIFE CYCLE COST MANAGEMENT IN NATO 


1.3.1 Life Cycle Cost Management Purpose 


Life cycle cost management (Reference: Life Cycle Management in NATO. A report to CNAD, edition 2, 
2002) includes the processes required to determine which resources (people, equipment, services, material 
etc.) and what quantities of each should be used to perform project/system activities, develop an estimate, 
of the associated cost and allocate them to individual work items. These processes are aimed to estimate 
system life cycle cost for decision making and budget allocation and to ensure that the system activities 
are performed within the approved budget and according to the operational requirements fixed. 


Figure 1-1: NATO Life Cycle Cost Management. 


1.3.2 Life Cycle Cost Management Activities 
¢ COST PLANNING 


¢ Develop the life cycle cost management plan and define the procedure by which the data are 
collected and managed. 


¢ Define the procedure for cost baseline allocation, change and configuration control. 


¢ Define the cost breakdown structure (CBS) that will be used throughout the system life cycle. 


¢ COST ESTIMATING 
e Identify the activities related to the CBS and estimate the associated cost. 


¢ Establish a cost baseline for measuring life cycle cost performance improvement. 


¢ COST BUDGETING 


¢ Allocate cost estimates to individual work items (related to the financial policy) in order to initiate 
the procedure for the allocation of the budget. 


+ Monitor and record cost performance" 


In this context cost performance means the monitoring and control of the cost estimate against the actuals. 
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¢ COST ASSESSMENT AND CONTROL 
e Detect cost variances from baseline. 


e Assess the “why” of both positive and negative variances and their impact (level of risk) on LCC 
and the allocated budget. 


e Ensure that all appropriate changes are recorded in the cost baseline. 


e Initiate the change request procedure for the allocation of budget. 


1.4 OBJECTIVES OF SAS-054 


There are many methods and models available to conduct life cycle cost analysis. It was important to 
understand the applicability and boundaries of each method and model in order to recommend and use 
them appropriately when conducting life cycle cost analysis. 


The core objective of the SAS-054 task group was to understand NATO and PfP nations’ methods and 
models and promulgate good practice. The primary focus was on methods and models that were developed 
within member nations. However, it was pertinent to recognise that commercial methods and models form 
an important and integral part of the toolset available to cost estimators and analysts. In order to meet the 
above aspiration the following objectives for the SAS-054 Task Group were defined as: 


¢ Review of cost forecasting models. 
¢ Review of cost forecasting methods. 


¢ Guidelines for the collection and understanding of cost related data for national and multi-national 
programmes. 


¢ Treatment of uncertainty and risk. 
¢ Guidelines for multi-national programmes. 
e Awareness of new developments in methods and models. 
¢ Review and possible enhancement of the NATO generic cost breakdown structure. 
The final deliverable of the study is a technical report containing the outputs from the objectives above. 


It was also expected that the SAS-054 task group will implement additional mechanisms for presenting the 
results. This could take the form of a symposium, lecture series, workshops or other formats. 


The task group was initiated in Spring 2004 and submitted its final report to the SAS Panel at the Fall 
2006 meeting. 


1.5 SAS-054 STUDY APPROACH 


At the initial meeting a matrix was developed for capturing the methods and models used by all the 
participating nations. This comprised a number of questions (see Table 1-1) that were applicable to each of 
the NATO Phased Armaments Programming System (PAPS) (The handbook was published in February 
1989 as AAP-20 (Allied Administrative Publication) by the Defence Support Division of the NATO 
International Staff). 


The responses from the questionnaire were used to understand and debate the issues that were relevant to 
life cycle costing. In this way information was captured in a cohesive and auditable manner. Each nation 
was then allocated a task that could be presented at a collective meeting and thoroughly discussed to gain 
a clear understanding of the topic. The completed matrix for each nation is given at Annex A. 
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Table 1-1: Summary Matrix Questions for Each Participating Nation 


Question 


Description 


Indicate the type of costing studies 
required. 


Nations were to state the reason for undertaking the cost study 
and show what the output was used for. 


Description of the process or national 
guideline to be followed — 
how are we going to do this? 


Nations were to state if there is a national guideline or process to 
be followed. 


What methods are used? 


Nations were to state the methods (Reference: Chapter 4) used at 
the different phases of the life cycle. 


What models are available? — 
Commercial models. 


State the commercial off the shelf (COTS) models (Reference: 
Chapter 5) used to achieve the required costing outputs. 


What models are available? — 
In-house developed models. 


List any formal models (Reference: Chapter 5) that had been 
developed in-house. Ideally these models would have been 
verified and validated as being “fit for purpose”. 


Requirements to apply national 
guideline. 


State if mandated by national government approval authorities or 
by departmental procedures. 


Requirements to apply methods. 


State if mandated by the national government approval 
authorities or by departmental procedures. 


Requirements to apply models. 


State if mandated by the national government approval 
authorities or by departmental procedures. 


Restrictions on applicable methods 
or models depending on the goal. 


State any barrier that may cause the method or model to be either 
unworkable or to rely on subjective judgement instead of data. 


How can data be collected? 


State if any automated system is used for collecting data 
(Reference: Chapter 6). 


How is risk and uncertainty 
considered? 


State the method used to identify, collect and analyse risk and 
uncertainty data (Reference: Chapter 7). 


What models and tools are available 
to assess uncertainty? 


State the commercial or in-house models used to produce 
stochastic or deterministic results through Monte Carlo 
modelling. 


What models and tools are available 
for risk analysis? 


State the commercial or in-house models used to collect and 
manage risk information. 


Requirements to apply risk 
methodology. 


State if risk assessment methodology is mandated by national 
government approval authority or departmental procedures. 
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1.6 IMPLEMENTATION OF THE GUIDELINE 


The use of life cycle costing should be encouraged both within government departments and defence 
contractors. Therefore, the overall objective of this report is to produce a guideline that can be used by 
NATO and PfP nations, defence contractors and equipment suppliers in both national and multi-national 
programmes. To achieve this, the implementation process for the guideline may include: 


e Advertising and publishing the guideline. 


¢ Developing some examples to demonstrate the life cycle costing process that can be adopted at 
the various stages of the product life cycle. 


¢ Organising technical seminars, lecture series and workshops. 


¢ Develop an improved NATO Standardisation Agreement (STANAG) on life cycle costing. 
However, this would have to explicitly describe in detail how it should be conducted and what is 
realistically achievable at each stage of the life cycle. 


The above list is not exhaustive and other implementation approaches may be considered. However, to be 
successfully conducted by NATO and PfP nations and equipment suppliers it will be essential to 
disseminate the life cycle costing process and techniques together with worked examples in order to gain 
the support of everyone. 


During the study, SAS-054 contacted the life cycle management group (AC/327). One of the tasks of this 
group is the responsibility for the development of NATO policies (e.g. STANAGs) related to life cycle 
management. In December 2005, the AC/327 subgroup B working group 3 was established. This working 
group is tasked to develop guidance on the application and implementation of NATO policy relating to life 
cycle costing in support of the NATO System Life Cycle Management (SLCM) policy (Reference. 
C-M(2005)0108-AS1 dated January 2006). 


The working group (AC/327) is the official and proper body to implement the RTO SAS-054 guideline 
within NATO and PfP nations. 


Further details on the work of AC/327 subgroup B Working Group 3 can found in their Terms of 
Reference (Reference: PFP(AC/327-SG/B)D(2005)0003 dated December 2005). 


1.7 STRUCTURE OF THE REPORT 

In addition to this introduction, the structure of the report comprises the following chapters. 

Chapter 2: Describes the role of life cycle costing in the decision making process. It examines the 
scope to be considered, its limitations and purpose in terms of budget planning, option 


analysis and cost reduction. 


Chapter 3: Provides a detailed insight into life cycle costing activities across each of the NATO 


PAPS phases. 

Chapter 4: Gives details on the life cycle costing methodologies that can be employed during each of 
the NATO PAPS phases. 

Chapter 5: Discusses life cycle costing models in terms of what is available, the appropriate 


applications and a brief on current models employed by NATO and PfP nations in 
conducting life cycle costing and subsequent cost analysis. 
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Chapter 6: 


Chapter 7: 


Chapter 8: 


Chapter 9: 


Chapter 10: 


Chapter 11: 


Chapter 12: 


Annex A: 


Annex B: 


Annex C: 


Annex D: 


Annex E: 


Discusses the issues and considerations found when examining each nation’s processes for 
data collection to support the life cycle costing activities. 


Provides the basis for the measurement of uncertainty and risk and how this should be 
used in life cycle costing and subsequent cost analysis. 


Considers any other life cycle costing issues and considerations that have been identified 
during the study. 


Highlights new developments in life cycle costing and cost analysis. 


Discusses any enhancement to the SAS-028 cost breakdown structure as a result of this 
study. 


Provides a summary of the findings and conclusions. 

Provides the recommendations. 

Provides the completed matrix completed by each nation. 

Provides examples of data forms for capturing the costs at the production phase. 
Provides examples of typical life cycle cost questionnaires. 

Provides a list of life cycle cost definitions. 


PAPS milestone definitions. 
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Chapter 2 - THE ROLE OF LIFE CYCLE COSTING 


2.1 GENERAL 


The use of life cycle costing should, in each phase of the programme, support the process by which 
managers can make the best decisions on options presented to them. These options may include evaluation 
of future expenditure, comparison between alternative solutions, management of existing budgets, options 
for procurement and evaluation of cost reduction opportunities. Life cycle costing is also used for 
affordability assessment and determining the cost drivers associated with the Key Performance Indicators 
or Key User Requirements. 


Life cycle costing must be used as a benchmark against which options can be measured for ‘value for 
money’ during the acquisition/production and in-services phases. However, it must be appreciated that the 
greatest opportunities to reduce life cycle costs usually occur during the early phases of the programme 
(as shown in Figure 2-1). It follows therefore that life cycle costing is used as a decision and optimisation 
criterion in the search of the best compromise between performance, cost and time. 


100 


80 (eee eee 
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60 -peeeeeeeaee 
alee es Cost incurred 


% of LCC 


40 - 


20 | 


Life Cycle 


Figure 2-1: Traditional LCC Committed versus Incurred Cost Curve. 


Life cycle costing should not be considered as a one-off task, but should be recognised as an ongoing 
activity throughout the life cycle to evaluate all programme changes and exploit cost saving opportunities. 


Although this report focuses on the importance of conducting life cycle cost analysis, it should be recognised 
that there are limitations of such an analysis. Some of the limitations are (Reference: LCC Tutorial by 
Paul Barringer and David Weber): 


e Life cycle costing is not an exact science. A life cycle cost analysis does not provide an exact 
number of the costs; it merely gives an insight in the major cost factors and an insight into the 
magnitude of the costs. 


e The life cycle cost estimate is only an estimate. Estimates can never be more accurate than the 
inputs and the inputs are often estimates themselves or expert opinions. 
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e Life cycle cost models require volumes of data and only a few handfuls of data is likely to exist 
when conducting the estimate. Therefore many assumptions have to be made. The life cycle cost 
estimate therefore only counts given the assumptions used. If one of the assumptions changes, it is 
possible that the cost estimate will change too. 


e Life cycle cost results are used for several purposes and, in some instances, are not compatible. 
For example, the life cycle cost used for a comparison or a trade-off study may not always be 
suitable for budgeting purposes. 


These limitations should be carefully considered when conducting a life cycle cost analysis. 


2.2 PHASES AND THE USE OF LIFE CYCLE COSTING 


It is recognised that individual nations may use their own nomenclature for these early phases (e.g. user 
requirement, system requirement, etc.) and may conduct their own pre-feasibility or early conceptual work 
to assess the level of their capability gap. For clarity and consistency the NATO nomenclature has been 
used throughout. However, the processes and techniques described here are equally applicable to national 
and multi-national programmes. 


Early in the project life cycle, studies need to address the capability gap, the numbers of equipment or 
platforms required and the technologies that can help to fill the gap at lowest cost. This requires a 
‘strategic’ approach that can provide a capability to look at the ‘big picture’. At this phase in the life cycle 
it is unlikely that the costs can be identified in a great deal of detail, rather an understanding of the 
holistic’ values (i.e. the whole is more than the sum of its parts) in terms of the primary cost breakdown 
structure elements and the uncertainty surrounding these figures is required. The level of life cycle costing 
at this phase will support the NATO MND (Mission Need Document) and ONST (Outline NATO Staff 
Target). It is important to recognise in these early phases that only broad estimates or a range of estimates 
will be available — it is more important to ensure that they are as complete as possible (e.g. nothing large is 
missing). 


Once the NST (NATO Staff Target) has been developed, the focus turns to the performance, cost and time 
envelope of various options that will meet the NST. Forecasts of the likely life cycle costs for new 
equipment(s) and platform(s) are needed so that the cost breakdown structure can be developed and 
extended to reflect the acquired knowledge of the expected system characteristics and associated costs. 
The life cycle costs at this phase will support the NSR (NATO Staff Requirement) by providing 
reasonably accurate estimates of development and production costs. However, due to the likely lack of 
design data the in-service costs will be more uncertain. During the project definition phase the usage 
patterns and system design will mature to provide a much improved basis for establishing more accurate 
in-service costs. 


When the preferred options are identified, industry is generally asked to provide information and compete 
for its supply. Assessments of the bids are conducted on a life cycle cost basis and need to address all the 
economic and financial requirements set out by each nation. At this stage the cost breakdown structure 
should be fully developed such that all the cost elements are identified. 


For in-service equipment a forecast of the costs for the remaining life is required. This will assist in any 
budget adjustment studies and provide a realistic baseline upon which to measure and compare with the 
effect of change due to utilisation, incremental updates, overhauls or even the procurement of new 
equipment. 


' The literal meaning of “holistic” is that all the properties of a given system cannot be determined or explained by the sum of 
its component parts alone. Instead, the system as a whole determines in an important way how the parts behave. 
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In summary, it is not possible or desirable to collect and analyse information at the same level of detail 
throughout the life cycle although there should be a common thread in terms of programme phases, 
cost breakdown structure grouping and resource consumption. What should be seen is a life cycle cost 
estimate that evolves, in terms of detail, as the programmes progresses through the different phases. 


Further discussion on the processes and methods applicable to each phase of the project life cycle is given 
at Chapter 3. 


2.3 APPROACH TO LIFE CYCLE COSTING 


Prior to any costing activity it is essential to define what is to be estimated and understand what the 
estimates will be used for (e.g. setting budgets, options evaluation, pricing, etc.). The system under 
consideration could range from a large turnkey project (e.g. a major capital investment including buildings 
and infrastructure), a stand-alone system (e.g. individual platforms such as a ship, aircraft or tank), or a 
worldwide application (e.g. theatre(s) of operation and use). The approach to be adopted needs to be 
tailored to suit the questions to be answered, the costing requirements and the availability of suitable data. 


With some variation (to the level of detail), the same basic approach to life cycle costing can be applied to 
all projects regardless of their specifications. This approach encompasses the following steps that are more 
fully described in the subsequent sub-sections. The steps are: 


e Define the aims and objectives of the study. 

e Establish the programme content, the costing boundary and the assumptions for the study. 

¢ Develop the structure of the life cycle cost framework. 

e Establish the data and populate the life cycle cost framework. 
Once the scope of the study has been established the overall cost estimating process can commence as 
shown in Figure 2-2. It may be necessary to undertake several iterations following the first set of results 


due to the availability of more data, clarification of the assumptions or just general refinement. 
The process is completed with the presentation of the results, assumptions and financial implications. 


Data Sources Generic NATO Assumptions 
: CBS 
Parametric General 
Analogy All Line Items Specific 
Bayesian Financial 
Detailed design Programme 
Industry data 
Reliability data 
Loaisti 
\ Colic ae ) Life Cycle Cost Risk 


Model 


Quantified 
Risk Register 
Optimism Bias 


Spreadsheet 


Proprietary 
Detailed Level 


Life Cycle Cost 
Output 


Figure 2-2: Generic Life Cycle Cost Estimating Process. 
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2.3.1. Define the Aims and Objectives 


The aim and the objectives of the study will have a major impact on the way a study is conducted. 
A different type of question will result in a different way of conducting the study. This will often be 
implicit in the type of study being undertaken, but it needs to be clearly and unambiguously defined if the 
life cycle cost study is to provide useful and meaningful results. An overview of the type of studies to be 
conducted can be found in Section 2.6. Early procurement phases will concentrate on assessing 
affordability, option analysis, etc., when the level of data to support a life cycle cost analysis is sparse. It is 
essential therefore to undertake several iterations of the process due to clarification of the assumptions and 
general refinement of the data. This stage would also include the definition of the systems and/or options 
to be studied. 


2.3.2 Establish the Costing Boundary and Assumptions 


The Costing Boundary defines exactly what cost elements will be included in the study and the level of 
detail in which they will be considered. The level of detail of the study is also dependent of some external 
factors, like the maximum duration of the study, the financial means available to conduct the study, 
the availability of qualified personnel to conduct the study, the availability of experts to provide 
information and the availability of data. 


There are three boundaries to consider. The first boundary relates to the definition of the system itself 
particularly the elements for costing. It is worth describing the total system and then agreeing with the 
stakeholders those elements that are outside the scope of the study. In this way a clear picture is drawn 
which helps to avoid any confusion later in the study. 


The second boundary definition addresses the timescale aspects so as to establish which programme 
phases have to be included, e.g. phased procurement, phased implementation, incremental build standards, 
in-service date and the likely operating life of the proposed system (see Sub-section 2.8.3 for details of the 
time period consideration to be made). 


The third and last boundary consideration defines what will be included within the scope of the study. 
This can range from the simple cost of buying a piece of equipment to the total cost to the government or 
industry of developing, procuring, operating, supporting and disposing of a complete range of such 
equipment. 


To ensure completeness, a full list of all possible cost elements should be first drawn up which covers all 
phases in the project life cycle. These cost elements should then be reduced by elimination of those falling 
outside the specified cost element boundary. This costing framework should be based on the NATO 
generic cost breakdown structure (Reference: RTO-TR-058 report). In Chapter 10 some enhancements to 
this cost breakdown structure are suggested. 


Figure 2-3 shows an example (taken from ANEP-41) on how the costing boundary expands firstly from 
just considering an equipment purchase (equipment buy) to the involvement in a development and 
production programme (procurement programme) and then into inclusion of the in-service costs. Further 
costs such as capability upgrades, etc., are included if the intention is to provide a forecast of future cost 
expenditure over the life of the programme. The costing of the final element shown is dependent upon 
each national requirement. Typically, an apportionment based on the expected use is agreed with the 
respective stakeholders and is added to the cost estimates for completeness. 
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ATO Ship Life Cycle Cost (LCC) Hierarchy 
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Figure 2-3: Typical Life Cycle Cost Boundary. 


Whenever an estimate is undertaken, it is necessary to create a series of statements that define the 
conditions upon which the estimate will be based. When conditions are directed upon the estimator, 
they become the ground rules by which the estimate will be conducted. In the absence of a firm ground 
rule the estimator can define assumptions. Because of the potentially significant cost implications of key 
assumptions and ground rules, it is good practice to undertake a sensitivity analysis of them. 


It is essential that all the data, ground rules and assumptions are captured and recorded so that there is a 
complete audit trail to the estimate. There is no NATO document that reflects this requirement. The next 
sub-sections describe how the UK and the US capture and record this. 


2.3.2.1. UK Master Data and Assumptions List 


The UK Ministry of Defence use and recognise a MDAL (Master Data and Assumption List) (Reference: 
www.ams.mod.uk). This is a comprehensive document that records all the data used in compiling the 
estimate and all the respective stakeholders are expected to sign that they agree with the data and 
assumptions recorded in the document. The process can be considered as laborious and time-consuming 
(but is essential) and in rapidly changing studies (e.g. pre-feasibility and concept) the requirement for a 
fully completed MDAL becomes cumbersome and constraining. This is overcome through the 
development of an abbreviated form of MDAL which may only be a few pages, but will contain all the 
salient points. This will be developed by the life cycle cost analyst and presented to the stakeholders for 
agreement. Developing the document in this manner ensures that the estimates can be produced rapidly 
and consistently to support option analysis and affordability assessment studies. 
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US Cost Analysis Requirements Description 


ORGANIZATION 


In the USA a process is adopted to establish the costing boundary; this provides the basis for the cost 
estimate. For major acquisition programmes, the CARD (Cost Analysis Requirements Description) is used 
to formally describe the acquisition programme (and the system itself) for the purposes of preparing both 
the programme office cost estimate and the US Department of Defense Component’ cost position, 
if applicable and the OSD CAIG independent cost estimate’. 


The CARD is prepared by the programme office and approved by the DoD Component Programme 
Executive Officer (PEO). For joint programmes, the CARD includes the common programme agreed to by 
all participating DoD Components as well as all unique programme requirements of the participating DoD 
Components’. 


The CARD typically provides both narratives and tabular data, roughly following the following outline: 


All ground rules and assumptions to be used in developing the cost estimates. 
System description and characteristics. 

e System work breakdown structure. 

e Detailed technical and physical description. 

e Subsystem descriptions, as appropriate. 

¢« Technology maturity levels of critical components. 

System quality factors. 

Reliability/Maintainability/A vailability. 

Project Managers’ assessment of programme risk and risk mitigation measure. 


System operational concept. 
e« Organisational/unit structure. 


e Basing and deployment description (peacetime, contingency, and wartime). 


System support concept. 

e System logistics concept. 

¢ Hardware maintenance and support concept. 
¢ Software support concept. 


e System training concept. 

Time-phased system quantity requirements. 

System manpower requirements. 

System activity rates (OPTEMPO or similar information). 
System milestone schedule. 


Acquisition plan or strategy. 


2 


A “Component” is a military department or a defence agency. 


> DoD Instruction 5000.2, Enclosure 3 specifies that for major defence acquisition programmes the CARD will be provided in 
support of major milestone decision points. 


* DoD 5000.4-M, DoD Cost Analysis Guidance and Procedures, Chapter 1 provides further guidelines for the preparation of the 
CARD. 
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For each topic listed above, the CARD should provide information and data for the programme to be 
costed. In addition, the CARD should include quantitative comparisons between the proposed system and 
a predecessor and/or reference system for the major topics, as much as possible. A reference system is a 
currently operational or pre-existing system with a mission similar to that of the proposed system. It is 
often the system being replaced or augmented by the new acquisition. For a programme that is a major 
upgrade to an existing weapon platform, such as an avionics replacement for an operational aircraft, the 
new system would be the platform as equipped with the upgrade, and the reference system would be the 
platform as equipped prior to the upgrade. 


Clearly, much of the information needed for the CARD is often available in other programme documents. 
The CARD should stand-alone as a readable document, but can make liberal use of appropriate references 
to the source documents to minimise redundancy and effort. In such cases, the CARD should briefly 
summarise the information pertinent to cost in the appropriate section of the CARD, and provide a 
reference to the source document. The source documents should be readily available to the programme 
office and independent cost estimating teams, or alternatively can be provided as an appendix to the 
CARD. Figure 2-4 illustrates the process. 


Develop Approach and Scope 


Cost 
Ground rules & assumptions Analysis 
System/Subsystem Descriptions CARD or Requirements 
Program Milestone Schedule CARD-like | Description 


Acquisition Plan or Strategy 
System Operating Concept 
Support Concepts (Logistics, Training) 


document 


Prepare Estimate 

= Select methods/models 

= Collect, validate & adjust data 

= Estimate costs 

= Assess risk and sensitivity 
Document and present results 


Coordination 


- Program Office, Users, 
Reviewing HQ, etc 

- Engineers, Logisticians, 
Financial Management, etc 

- Form Cost IPT as needed 


Figure 2-4: USA Cost Analysis Process. 


It is recommended that all the assumptions used and recorded be questioned by an independent 
technical team. 


2.3.3. Develop the Structure of the Life Cycle Cost Framework 


Having established the costing boundary this will set the requirements for the life cycle cost framework 
(e.g. the implementation could be a spreadsheet or more advanced modelling techniques). The costing 
framework should take account of both the immediate needs of the current phase and also to be adaptable 
to the developing needs of later phases. 


The structure of the framework will be based on the requirements previously discussed at Sub-sections 
2.3.1 and 2.3.2. A typical outline for a generic life cycle cost model is shown at Figure 2-5. 
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Develop Input 
==> a pattt Parameters/Screens 


Output Screen/Curve 


Figure 2-5: Example of a Typical Life Cycle Cost Framework Structure. 


The figure shows the framework broken down into a number of areas. On the left (in green) would be the 
cost breakdown structure reflecting the system, any specific options relating to the system and details of 
the programme timescale. On the top (in blue) would be the programme documentation relating to 
procurement strategy, how the system will be deployed in operational and peacetime use and how it will 
be supported in these environments. The ILS (Integrated Logistic Support) and the LSA (Logistic Support 
Analysis) inputs would support the understanding of the proposed deployment. At the bottom (in yellow) 
would be the ground rules and assumptions. These would be recorded in a document and the information 
would be used to populate the areas of the model where no hard data was available. On the right hand side, 
the risk issues could be included within the model so as to obtain a ‘risk adjusted’ cost. To avoid an over 
complicated implementation it may be necessary to develop some suitable input and output screens to 
assist the user. 


It is essential that all life cycle cost models are robustly tested and validated to ensure the correct operation 
of the equations in relation to the input attributes. 


2.3.4 Establish the Data and Populate the Life Cycle Cost Framework 


Once the structure of the life cycle cost framework has been established, the cost breakdown structure 
needs to be populated. The cost breakdown structure will comprise a number of cost elements. These cost 
elements will need to be estimated. The method of establishing the estimate will depend on the availability 
of the data. An overview of the methods to estimate costs can be found at Chapter 4. 


Typical methods employed to gather data are: 


Market Survey 


This is usually good for gathering technical data, but limited in obtaining prices. Any costs obtained are 
likely to have a large margin of error and will have little or no substantiation. 


Industrial Visit 


This will produce information and data on the product, but it is unlikely that the companies will provide 
anything more than broad order cost and again with no substantiation. 
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Data for Direct Use Within the Study 


This would be actual costs or estimates directly relating to the breakdown of the system. Actual equipment 
procurement costs, published data for equipment and services, etc., could fall directly into this category. 
A considerable element of these costs will be based on current records and may be held by specialist cost 
estimating departments or consultancies. 


Data on Analogous Systems 


This could be low level, technical and other data on the system under consideration for use in parametric 
or other modelling methods to derive the required information. Size, weight, complexity, reliability could 
be examples of this kind of data. 


Data from Logistic Analysis 


This would be information and data from ILS (Integrated Logistic Support) studies to provide data on 
component reliability, maintainability and supply chain information. The cost estimator should be aware 
of the following essential ILS information to support the life cycle costing (accepting that some of these 
will not be available in the early phases of the programme). 


The work frame and interfaces of the ILS Programme for a generic defence system are described in an ISP 
(Integrated Support Plan). This is the main ILS management plan. Specific plans are applicable to the ILS 
Programme, for example (but not exclusively) the following: 


e LSAP (Logistic Support Analysis Plan). 

¢ Reliability and Maintainability Programme Plan. 

¢ LCC (Life Cycle Cost) Plan. 

¢ MANPRINT (Manpower and Personnel Integration) Plan. 

e LORA (Level of Repair Analysis) Plan. 
The LSA (Logistic Support Analysis) constitutes the on-going iterative analytical process that is employed 
to: 

e Assess a given or assumed configuration in terms of its inherent supportability characteristics. 


e Identify the ultimate support requirements for that configuration. 


It should be emphasised that the results from the LSA contribute to life cycle cost in two distinct ways: 


1) The application of the analysis provides data from which life cycle costs are derived; and 


2) LSA results contribute to the reduction of life cycle costs by indicating the support cost drivers for 
the system and thus providing feedback for the ILS design influence. 


The life cycle cost evaluation (see Figure 2-6) is based on the life cycle cost modelling effort, which in 


turn takes advantage of the Use Study to establish the full range of operational parameters necessary for 
the calculation of realistic costs. 
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USE 
STUDY 


SYSTEM 


FUNCTIONAL PREPARE 
(ARCHI TECTUR Lcc 
CONCEPTS MODELS 
MODELS 
APPROVAL 
Approved Cost data from 
models baseline COMPARATIVE 
COST DATA comparison ANALYSIS (e.g. 
ASSUMPTIONS system pcr edir nant 
(LABOUR COSTS K or 
ADMIN COSTS ; Se equivalent ) 
ETC., |.A.W. Manpower 
APPROVED TASK ANALYSIS 


MODELS) (e.g. MIL STD 
1388 TASK 401 
or equivalent ) 


PREPARE 
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REPORT 


Figure 2-6: Process Flow for Determining LCC Support to Logistic Analysis. 


Additional inputs to the life cycle cost evaluation can come from the execution of LSA Task 203 
(Comparative Analysis) and Task 401 (Task Analysis). 


The purpose of Task 203 is to identify, by using comparative elements, supportability problems that have 
arisen in the employment of systems already in service and to identify supportability, cost and readiness 
drivers that must be considered as critical in the comparative system design. 


As regards Task 401, its purpose is to focus on potential critical support elements and to influence the 
system design in order to reduce the impact of critical support elements. 


The LSA task results are documented in the LSAR (Logistic Support Analysis Record) to identify and 
develop logistic support resources. 


The LSAR database is initiated early in a programme to capture logistic requirements, identify functional 
logistic breakdown structure and document the initial results of the LSA tasks. Prior to the production 
phase, the LSAR database is used to: identify the complete logistic breakdown structure; control logistic 
performance of the system; identify operating and maintenance tasks, logistic support resources and 
transportability characteristics. During the production phase, the LSAR database is used to update the 
identification data due to configuration changes, document the LSA task updates and document the results 
of validation of the technical documentation. During the in-service phase, the LSAR database is used to 
capture R&M (Reliability and Maintainability) data related to the use and support of the system, update 
logistic data due to configuration changes, manage obsolescence and the effect from any maintenance 
organisation changes. 
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Figure 2-6 illustrates how the information flow from various documents and studies are combined in order 
to provide the life cycle cost to support logistic analysis studies. 


2.4 CONDUCTING THE COST ANALYSIS 


The level of analysis required by different studies varies considerably. In some circumstances, simple 
accountancy calculations involving discounted cash flow may be all that is required. Cost analysis would 
traditionally include the testing of parameters and assumptions by means of sensitivity analysis. Testing of 
alternative assumptions by means of “what if’ analyses should also be conducted. It is essential that any 
life cycle cost model has the ability to support these types of analyses so that the decision-makers have a 
full understanding of the costs and the financial implications. 


Quantitative cost risk analysis may also be employed to either gain more insight into specific risk areas or 
to evaluate a programme’s overall exposure to risk. A key feature of this approach to analysis is that it 
should provide forecasts and insights based on complex combinations of risks and uncertainties, as could 
occur, for example on major projects. Commonly used techniques include Monte Carlo analysis and 
System dynamics. Figure 2-7 shows the benefit of conducting cost risk analysis in addition to the more 
traditional accountancy methods of establishing financial outcomes. 


Wis Budget for all Budget including risk- 
90% planned activities based contingencies 
T 


80% 
70% 
60% 


Variance of budgets Full cost model, 
50% for planned tasks including all risks 


40% 
30% 
20% 
10% 


2,000,000 2,100,000 2,200,000 2,300,000 2,400,000 
Cost 


Figure 2-7: Cost Risk Analysis Outputs Example. 


The first curve (green) represents the uncertainty in the cost estimate arising from uncertainty in the data 
and methodology employed in developing the estimate. The second curve (red) represents the extent of the 
risk in the programme arising from the relevant risks recorded in the programme risk register. 
In examining both curves the decision maker can establish the level of acceptable financial risk dependent 
upon the confidence in the cost estimate. More details on uncertainty and risk analysis can be found in 
Chapter 7. 


25 PRESENTATION OF THE LIFE CYCLE COST RESULTS 


Summaries of the life cycle cost outputs and the underlying assumptions should be discussed with the 
stakeholders at the earliest opportunity. This will allow for changes and refinements to be incorporated 
prior to the issue of a final report. The objective at all times is to ensure the life cycle cost study is 
meaningful and fully meets all the stakeholders’ needs. 


The results of cost studies can be presented in a wide range of tabular and graphical forms. The favour is 
to include graphical presentations of the results wherever possible. This enables the widest possible 
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ORGANIZATION 


audience to have a clear picture of the overall results while retaining the detailed tabular presentations for 
those that require them. 


Two common form of graphical presentation (the spend profile and cost allocation pie chart) are shown as 
Figures 2-8 and 2-9. These figures indicate costs at a high level, but can also be used to present a more 
detailed level as required. For presentation purposes these costs have been truncated at Financial Year 


(FY) 18. 
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Figure 2-8: Example of a Baseline Life Cycle Cost Spend Profile. 


Production 
29% 


Development Operations 
22% 19% 


Figure 2-9: Example of a Life Cycle Cost Allocation. 


Note: The cost allocation percentage shown in the example above should not be considered as being 
representative of all life cycle cost estimates. 
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The product of the life cycle cost study should be a report incorporating the results and conclusions as well 
as a presentation on the basis of those results. It should include a full definition of the aims and conduct of 
the study, the definitions of the options studied, the costing boundary considered and the assumptions 
underlying the cost elements. This work should also be continuous to support the life cycle management 
of the programme. 


The figures above represent single point estimates with no consideration to the presentation of uncertainty 
and risk. Figure 2-10 presents a recommended approach for communicating results of a life cycle cost 
estimate to senior decision makers.° The top line shows a three point range of estimates, and conveys the 
idea that a cost estimate is not a single number, but rather a continuum or distribution of possible values. 


Low High 
Estimate Baseline Estimate Estimate 
585M 715M 895M 
| ! | 
500M 600M 700M 800M 900M 


Cost Growth Factor, Sensitivity Analysis, or Risk Register from the UK 


Risk and Uncertainty Analysis Using Monte Carlo Simulation 


Underlying Assumptions or Scenarios 


Figure 2-10: Recommended Presentation of Cost Estimating Risk Analysis. 


Analysts can use one or more estimation techniques in performing risk and uncertainty analysis. Some of 
these are shown in the top two bars or sections of the figure. The bottom section, which should always be 
included in the presentation of the estimate, shows all of the assumptions or scenarios associated with the 
low, baseline, and high estimates. Including this section enables decision makers to see clearly the cost 
implications of events that can influence the outcome of an acquisition programme. 


This approach will lead to the establishment of a sound, well-structured methodology for the conduct of 
and presentation of life cycle cost estimates. 


2.6 FINANCIAL THRESHOLD 


Some nations have set a financial threshold for conducting life cycle costing studies. For example, 
in The Netherlands a life cycle cost analysis (as part of the defence materiel process) is performed for all 
projects with a total budget in excess of €5 M. Others have no limits, but are determined by: 


PLE: Ministry of Defence and Impossible Certainty, RAND, 2006, pages 84-86. 
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¢ The level of detail required. 
e The benefit the life cycle costing study will bring to the programme. 
e The rigor necessary to support the business/programme investment. 


e The likely effort to be expended. 


It is recommended that each nation sets its own threshold value and that it should be determined in 
terms of total programme cost, political requirements and timeliness. 


2.7 ORGANISATION AND OWNERSHIP OF THE LIFE CYCLE COST 


The life cycle cost analyst or manager should be the owner of the development of any life cycle cost 
estimate and the configuration management of the supporting documentation. 


The role of the life cycle cost analyst is to ensure the smooth running and facilitation of the total life cycle 
cost management process. In summary, the analyst should be responsible for: 


e Ensuring that an appropriate life cycle cost management plan is in place and updated where 
necessary. 


e Supporting the programme by providing robust and credible life cycle costs in a timely manner. 


e Ensuring that the life cycle costing process is appropriate, workable and supports the programme 
requirements. 


e  Eliciting life cycle cost information from both the government and contractor project teams. 


e Reviewing all assumptions and contractual change notices from the project teams and advising the 
programme manager of any points of issue. 


¢ Providing guidance and assistance for the cost risk analysis and associated reports. 


e Ensuring the smooth running and facilitation of the total life cycle cost management process, 
including the regular reporting procedures. 


To achieve the above in the most practicable, auditable and robust manner it may be necessary to conduct 
the life cycle cost analysis using multiple methods and/or independent experts. This will depend on the 
overall value of the likely programme costs and the level of robustness needed for the government 
approval process. 


2.8 TYPICAL APPLICATIONS OF LIFE CYCLE COSTING 


One of the principal objectives of life cycle costing is to reduce or control the life cycle cost by assessing 
the financial impacts of the decisions taken about the complete system. 


Three broad classes of applications rely on the output from life cycle costing and are discussed in detail 
below. These are: 


e Determining the forecast of future spending. 


e Examining comparisons between alternative solutions (e.g. alternative assets, design trade-off, 
supply chain analysis, etc.). 


e Supporting the tender evaluation process. 
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In all cases, the output of the life cycle costing provides information to support the decision making 
process. Note however, that cost is just one of many criteria that could influence the decision. Other 
criteria such as operational effectiveness, technical risk, political and industrial policy constraints, etc., 
also have to be considered in the decision making process and are sometimes more important than cost. 


2.8.1 Determine the Forecast of Future Spending — Defence Budget Planning 
Applications 


Budget planners are often confronted with choices between several distinct systems (e.g. aircraft or UAV 
or missiles; ships or forward bases, etc.). The life cycle cost estimate can help the decision process by 
addressing the following typical questions: 


¢ In consideration of long term planning applications (~10+ years): 


e¢ What will be the cost of the systems currently being designed (both in terms of money spent 
annually and of the number of service personnel required to man the systems)? 


e¢ What is the best ratio between money spent on investment (new systems or upgrades of 
“old ones’) and that spent in order to keep the readiness of currently available systems? 


¢ Inconsideration of short term planning applications (~next 1 to 4 years): 


e How many systems (or individual platforms) can we afford (now) and still maintain some 
flexibility in future budgets (considering their estimated in-service costs)? 


2.8.2. Examining Comparisons between Alternative Solutions 


Comparative studies are particularly valuable in the early stages of planning when the primary objective is to 
establish an efficient and economical course of action. Comparative studies are, actually, used throughout all 
phases of a system’s life cycle; they are also used in selecting in-service options such as in-house or 
contractor support. 


An analytical comparison of the operational effectiveness, suitability and life cycle cost of alternative 
programmes that satisfy established capability needs is referred to as AoA (Analysis of Alternatives). 
An AoA broadly examines multiple elements of programme alternatives including technical risk, design 
maturity and cost. AoAs are intended to: 


e Illuminate the risk, uncertainty and the relative advantages and disadvantages of the alternatives 
being considered. 


e Show the sensitivity of each alternative to possible changes in key assumptions. 


¢ Help decision makers in determining whether or not any of the proposed alternatives offer sufficient 
operational and/or economic benefit to be worth the cost. As a general rule, the preferred alternative 
is the alternative that provides the greatest amount of benefits in relation to its cost. 


2.8.3 Supporting the Tender Evaluation Process 


In the tender evaluation process the life cycle costs can be used to ensure that the contract award is made 
to the tenderer who offers a system that meets all technical and availability requirements at minimum life 
cycle cost. The cost of investment in reducing maintenance resources and the cost of lifetime support will 
be weighed against the cost of investment in the overall system. The resulting life cycle cost will therefore 
be beneficial to the overall tender evaluation process. 


To establish a cost-effective in-service phase it is essential to consider operating and maintenance issues at 


the same time as the procurement of the system. The life cycle cost from the evaluation process can often 
be used as a baseline for negotiation on contractor logistic support contracts. 
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The choice of a proper time period (system life) in the life cycle cost evaluation process must be 
considered. For example, many parameters can influence the selection of a well balanced time-period. 
Very often the total technical life of a system may not be the most appropriate time period for the life 
cycle cost evaluation. A shorter time-period takes more consideration of the initial acquisition costs, and a 
longer time-period takes more consideration of the recurrent ownership costs. 


Figure 2-11 presents the life cycle costs for two competing systems A (green) and B (red). The initial 
acquisition price for System B is less expensive than for System A. However, System A has lower annual 
cost for ownership than System B. At year 10 the cost-lines intersect, and after year 10 System A has a 
lower life cycle cost compared to System B. The example shows the complexity of choosing the “correct” 
time period to include in a tender evaluation. This example clearly demonstrates that the selection of a life 
cycle time period must be tailored and well balanced to fit its purpose. 
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—e SYSTEMB 
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Figure 2-11: Example of Time-Period Consideration. 


To ensure that all the tenders are impartially evaluated it is vital that the cost breakdown structure is 
defined by the procurement agency. This should include a definition of all the cost elements. Sufficient 
data on the likely use of the system should also be included in the request for quotation. This will improve 
the prospective supplier’s ability to independently assess and possibly improve their offer. 


It is recommended that a life cycle cost questionnaire is issued with the request for quotation so that the 
procurement agency can conduct an independent comparative life cycle cost evaluation on all the 
tenders. This will improve the understanding of the tender offer and provide a degree of credibility in 
the life cycle cost results. 


However, before the request for quotation is issued, it is important that all the preparatory work has been 


independently conducted and that the Key User Requirements are well balanced between functionality and 
likely costs. 
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A life cycle cost evaluation starts with the quality of the submitted tender data. In most cases it is 
necessary to iterate the process several times in order to obtain clarification and to explore opportunities 
for improvements. Figure 2-12 shows the tender evaluation process with specific feedback to the tenderers 
in order to exploit opportunities for cost reduction. The tender evaluation process is completed by 
documenting the LCC results in an evaluation report before the contract can be awarded. 


Tender Study Tender Data “Lead the Force” Request for Meeting with Tenderers update 
The most complete complementary — Tenderers Tender Data 
tenders are analysed Tender Data J 
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x @L 


Evaluation 1 


Feedback are sent 
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‘valuation 
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Meeting with Tenderers send an Evaluation 2 Evaluation report Contract 
Tenderers improved Tender 
Data 


Figure 2-12: LCC and the Tender Evaluation Process. 


2.9 ASSESSING AFFORDABILITY 


This is not common practice in all nations, but it is one that we recommend should be conducted. 


By definition, affordability can be considered as the degree to which the life cycle cost of an acquisition 
programme is in consonance with the long-range investment and force structure plans of national defence 
administrations. Affordability procedures establish the basis for fostering greater programme stability 
through the assessment of programme affordability and the determination of affordability constraints. 
In this context: 


e Inconsonance means delivering systems that meet the customer’s needs and budget. 
¢ Programme stability means working towards sustainable opportunities. 


e Assessment means creating a programme management strategy that guarantees programme 
viability. 


¢ Determination of affordability constraints means bringing affordability to the foreground to 
avoid misconceptions in management and engineering which may ultimately lead to unaffordable 
design solutions. 
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Affordability plays an important part in programme decisions throughout the life-cycle. Even before a 
programme is formally approved for initiation, affordability plays a key role in the identification of 
capability needs. Programme affordability is part of the process which balances cost versus performance 
and in establishing key performance parameters. 


The exact approach to the affordability assessment can vary, depending on the nature of the programme. 
However, in general, the assessment should address programme funding and manpower requirements over 
the whole life period. The assessment also should show how the projected funding and manpower fits 
within the overall plan for simultaneous programmes. The assessment should highlight those areas where 
the projected funding or manpower share exceeds historical averages, or where the projected funding or 
manpower exceeds zero real growth from the last year of the programming period. For the issues 
highlighted, the assessment should provide details as to how excess funding or manpower demands will be 
accommodated by reductions in other mission areas, or in other (e.g. non-modernisation) accounts. 
To illustrate this approach, this section provides a notional example of the type of analyses that could be 
incorporated in an affordability assessment. Although this example only addresses modernisation funding, 
the approach for manpower would be similar. 


In this hypothetical example, a major defence acquisition programme is nearing government approval. 
For discussion purposes, this programme arbitrarily is assumed to be a mobility programme. A first step in 
the programmes affordability assessment is to portray the projected annual modernisation funding 
(Research, Development, Test and Evaluation plus procurement, measured as total authority obligation) 
in constant dollars for the six-year programming period, and, in addition, for an additional twelve years 
beyond that. Similar funding streams for other acquisition programmes in the same mission area (in this 
example, mobility) also would be included. Figure 2-13 is a sample chart for this first step. 
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Figure 2-13: Sample Chart of Funding Streams by Programme. 


In this example, the acquisition programme nearing approval is labelled “Mobility MDAP #3.” Funding 
also is shown for the other modernisation programmes in the same mission area, consisting of three other 
major defence acquisition programmes, three other (Acquisition Category II) programmes, and one 
miscellaneous category for minor procurement. In this example, there appears to be a significant 
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modernisation bow wave beginning around 2014, which would then be subject to further analysis and 
discussion in the assessment. The term “bow wave” refers to a requirement for excess modernisation funds 
during a period beyond the programming period, resulting from acquisition decisions made earlier. 


The second step in this assessment is to portray the modernisation funding stratified by mission areas, 
rather than by individual programmes. Figure 2-14 shows a notional example of this second step. 
The choice of mission areas will vary depending upon circumstances. Clearly, an analysis by an individual 
Component® would portray funding only for applicable mission areas. Also, for a Component like the 
Army, where almost all of its modernisation funding is in a single mission area (Land Forces), the mission 
area should be further divided into more specialised categories (such as digitisation, helicopters, ground 
combat vehicles, etc.). 
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Figure 2-14: Sample Chart of Funding Streams by Mission Area. 


For this example, Figure 2-14 shows funding growth in three mission areas (space, missile defence, 
and mobility). What remains to be determined is whether this projected growth is realistically affordable 
relative to the Department’s Component’s most likely overall funding (top-line). The third step in this 
assessment is to portray annual modernisation funding compared to the Department’s Component actual or 
projected funding top-line, as shown in Figure 2-15. There are three distinct time periods considered in 
this figure. The first is a twelve-year historical period, the second is the six-year programming period, 
and the third is the twelve-year projection beyond the programming period. What this chart shows for this 
example is that the assumed mobility programmes are projected to require a significantly higher share of 
the Department’s Component funding in the years beyond the programming period. In such a 
circumstance, the Department’s Component would be expected to rationalise or justify this projected 
funding growth as realistic (by identifying offsets in modernisation for other lower priority mission areas, 
or perhaps by identifying savings in other accounts due to business process improvements or reforms). 


ae “Component” is a military department or a defence agency. 
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Figure 2-15: Sample Annual Modernisation Funding. 


2.10 SPECIAL REQUIREMENTS FOR MULTI-NATIONAL PROGRAMMES 


The life cycle cost studies for multi-national programmes follow the same principles as defined in the 
previous sub-sections. Nevertheless, there are some specifics that should be taken into account in terms of 
organisation, studies, model(s) and in the presentation of the results. 


2.10.1 Definitions 


Multi-national programmes involve at least two nations who have agreed upon the main principles of 
co-operation in a MoU (memorandum of understanding), or an equivalent arrangement, for one or several 
phases of the entire lifetime of that programme. 


2.10.2 Specifics of Multi-National Programmes 


2.10.2.1 Framework 


A programme becomes multi-national as soon as a MoU is agreed and signed between the participants. 
In the NATO environment, it is assumed that such a MoU would normally start with the feasibility phase, 
at the earliest’. This means that the life cycle cost studies undertaken during the first phases (Mission Need 
Evaluation and Pre-Feasibility phases at least) are done at a national level. 


2.10.2.2 Added Value 


A significant added value of a multi-national programme is the possibility of setting up common solutions 
for procurement and support. This results in reduced life cycle cost due to research for greater 
commonality across the participants. 


7 Reference AACP-1 Allied Acquisition Practices Publications. 
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Another mutual benefit of a NATO collaborative programme is the economic saving that accrues due to cost 
sharing. In a NATO collaborative programme, the principal area of cost sharing occurs in the common non- 
recurring cost elements and the operation of the international programme office. Cost shares may be 
determined or allocated to the participating nations in a number of ways (e.g. costs may be allocated based 
on quantities procured during production, capital outlay of the respective nations, or other agreed means). 


The co-operation of several nations within a multi-national programme may result in higher absolute costs 
— due to additional expenses for travel, accommodation, communication, information exchange, 
standardisation agreements, approval of procedures, etc., than the case of a national programme. 
Nevertheless, when sharing this higher total amount, the share per nation will usually be lower than for the 
same cost elements of a national programme thus resulting in overall programme savings from a national 
perspective. Some of these initial additional efforts such as developing a common methodological 
approach, standardisation agreements, etc., may be incurred only once and then be made available to 
future programmes at minimal additional cost. 


2.10.2.3 Taking into Account “juste retour’ (Fair Return on Investment) 


Although common solutions reduce an overall life cycle cost, it is obvious that the participants will require 
a balance of financial and technological demands against the industrial benefit that flow from the 
programme (“juste retour” or fair return). This last requirement (industrial benefit) should also be taken 
into account in the life cycle cost studies. 


This consideration implies the definition of a work sharing methodology to be utilised in future NATO 
programme cost estimating applications. The methodology requires an equitable distribution of work share 
for engineering, manufacturing and service related activity across the participating nations according to 
the capital outlay of the respective nations or as they otherwise may agree. The goal is to achieve an 
overall sharing balance through an optimum mix of development and management work, centrally 
procured major equipment, domestically procured miscellaneous material, land based facilities, 
and national construction activity. Additionally, offsets may be necessary in some cases and cost- 
effectiveness must be considered. 


An objective evaluation is necessary to determine the optimum mix (technical requirements met and cost- 
effectiveness within work sharing constraints) of centrally procured equipment. This process involves 
assessing bids from participating national industries to establish an optimal distribution based on technical, 
cost, and work sharing considerations. Due to the large number of possible combinations, a computer 
assisted evaluation model is helpful if not essential. 


2.10.3 Organisation 


2.10.3.1 Multi-National Management Structure Models 


There are basically three models for setting up a management structure for multi-national programmes in a 
NATO environment. These are: 


¢ The pilot nation model, where the responsibilities for the day-to-day management of the 
programme, under the direction of a joint steering committee are exercised by one of the 
participants on behalf of all. 


¢ The integrated model, where day-to-day management of the programme is secured either by an 
IPO (international programme office) under the authority of a joint steering committee or by an 
executive agency of a NATO organisation under the authority of a board of directors, having 
international status as a legal entity under a NATO charter in accordance with C-M(62)18°. 


3 C-M(62)18 is related to the NATO council resolution on regulations for NATO production and logistic organisation. 
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¢ The decentralised model, where the joint steering committee has minimum power of decision, 
serving mainly as a medium for co-ordination and information. 


2.10.3.2 Consequences on Life Cycle Costing Organisation 


In the case of the first and second models of multi-national management structures, the pilot nation or the 
IPO or the NATO agency could be responsible for the life cycle cost estimates and comparison of 
alternatives within a scope to be defined by the participants (see Sub-section 2.10.4). One or more 
participant(s) could perform a peer review including a verification and validation of the life cycle cost 
studies performed above. The basic principle is that these studies should be based on an agreed common 
life cycle cost model (see Sub-section 2.10.6). 


In the case of the third organisational model of multi-national management structure, the participants 
could agree on choosing one of them for the performance of the life cycle cost studies. Therefore the same 
organisation scheme as described above could be applied. 


2.10.4 Scope of the LCC Studies in Multi-National Programmes 
There are two main parts in undertaking life cycle cost studies for multi-national programmes. These are: 


¢ Life cycle costing related to national specifics. For example, government furnished equipment 
acquisition, national programme management team. 


e Life cycle costing related to the development of commonality part where different alternatives 
have to be compared and evaluated. 


The first part is mainly a national responsibility. Nevertheless, if the participants want the pilot nation or 
the IPO or the NATO agency to perform a complete study, then the outputs of the life cycle cost relating 
to the national specifics should be inserted into the common model. 


The second part is in the scope of the life cycle costing studies in multi-national programmes. Here it is 
necessary to identify the areas in which alternatives could be defined and assessed. Figure 2-16 below 
shows a process for the selection of the areas and the definition of the related scenario. 


Selection of the options 
to be assessed 


Figure 2-16: Identification of the Options to be Assessed. 
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2.10.5 Example 


The different options to be considered could therefore relate to the management of support during the 
in-service phase particularly over the maintenance activities and of the support equipment. The life cycle 
costs would therefore have to consider the effects of the following activities: 


¢ Common maintenance by participant nations. 

¢ National maintenance on behalf of other nations. 
¢ Common contractor logistic support. 

¢ NATO maintenance. 

¢ National maintenance. 


Based on the results from the life cycle costing the selection of the options to be pursued would be based 
on the respective savings, opportunities, operational policy, etc. 


2.10.6 Setting up a Common Life Cycle Cost Framework 
Setting-up a common life cycle cost framework includes the following steps: 
Step 1: The selection of the different models that will be used in common for the programme; 


Step 2: The definition of the Programme Cost Breakdown Structure (CBS) and the way of 
aggregating the results of the different models into one framework; 


Step 3: The definition of the assumptions to be applied for the LCC calculation and to be used for 
data collection; and 


Step 4: — The collection of the data. 


Step 1: Selection of Common Life Cycle Cost Models 


All interested parties in the life cycle cost calculation have to agree on the use of common models. 
This will ensure that all subsequent calculations can be compared on an equal basis. The models should be 
either programme specific or COTS (commercial off the shelf), as required by the need of the programme. 
The functionalities of such models must allow an easy interface with those models eventually defined at a 
national level. 


Step 2: Definition of a Common Cost Breakdown Structure and Output Aggregation 


Together with the selection of the models, a common CBS (cost breakdown structure) must be defined for 
the programme. This CBS should be based on the NATO generic cost breakdown structure (Reference: 
RTO-TR-058 report). It should also allow the insertion of the cost elements relating to national specifics if 
required (see Sub-section 2.10.4). 


The results from the different models selected in step one should be aggregated following the CBS defined 
above. Figure 2-17 shows this aggregation. 
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Possible Models Aggregation 


Figure 2-17: Models Output Aggregation. 


Step 3: | Assumptions (see also Sub-section 2.3.2) 


Once the models (with the way of aggregating their outputs) and CBS have been defined, the assumptions 
necessary to define the cost data should be commonly determined and agreed. The pilot nation or IPO or 
NATO agency should build an MDAL or CARD or equivalent where all the assumptions necessary to the 
definition of the cost elements are included. It is acceptable (and in some instances, inevitable), 
that different nations will use different assumptions. This should, however, be reflected in the MDAL, 
CARD or equivalent document. 


Great care must be particularly applied to the assumptions of deployment. It is likely that each nation may 
have slightly different deployment and supportability considerations than are usually reflected in the 
memorandum of understanding or similar arrangements. These should be taken into account in the 
MDAL, CARD or equivalent document while a consensus for a multi-national scenario is to be defined. 
In this way, optimised solutions can be evaluated to address both national and multi-national concerns. 


Step 4: Data Collection (see also Sub-section 2.3.4) 


Based on the agreed CBS and MDAL, CARD or equivalent, the data required to feed the models will have 
to be collected. The source of that data can vary between programmes, but it will generally include 
participating nations, industry and the pilot nation or IPO or NATO agency. 


2.10.7 Presentation of the Results: Currency Issues 


Generally, the cost data utilised as input to a multi-national programme is initially specified in the currency 
of the supplier nation. For non-domestic work and items subject to international influence of the participating 
nations, exchange rates between the national currencies are utilised as the currency conversion mechanism to 
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develop national cost estimates. Application of escalation formulae using indices from the supplier nation 
allows for changes due to national inflation prior to conversion to other currencies. 


With the advent of the European Union, more and more nations have adopted the Euro instead of their 
former national currencies and the Euro has become the reference currency for expression of multi- 
national cost estimates and cost targets. Such a reference currency may be useful for performing design 
trade-offs or pricing the average cost for comparison against an established cost target. 


Some past programmes have utilised Purchasing Power Parities (PPP) as the currency exchange 
mechanism for domestic work under the direct control of the respective nations, e.g. labour. PPP are rates 
of currency conversion that equalise the purchasing power of the different national currencies. Variations 
in PPP over time are mainly governed by the difference of inflation rates between countries. PPP vary 
slowly in contrast to currency exchange rates that are subject to market stresses. PPP may be useful when 
comparing costs between specific nations and may provide insight useful to planning purposes of the 
respective nations or for trend forecasting associated with NATO collaborative programme cost estimates. 


Despite the need for a reference currency and the issues of currency exchange that must be agreed upon, 
it is recommended that each nation apply its own cost model and applicable data (Cost Estimating 
Relationships, labour rates, etc.) to arrive at its national cost estimate. 

The choice of the currency should be made before the launch of the life cycle cost studies and the 
evolution of the exchange rates taken into account in the model. It is highly recommended that the life 


cycle cost analyst seeks economic advice that is pertinent to the particular programme to be evaluated. 


See also Sub-section 6.5.4 for more details on exchange rates and currency issues. 
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Chapter 3 —- LIFE CYCLE COSTING IN THE PAPS PHASES 


3.1 GENERAL 


Most nations use their own standards and nomenclature for describing the life cycle of a project. In multi- 
national projects and documents for use in various national and multi-national contexts, a common 
standard is needed. The Phased Armaments Programming System (PAPS) provides such a common 
standard, approved and accepted by NATO nations. For this reason, the PAPS definitions of the life cycle 
phases have been adopted in this report. Accordingly, this chapter is structured along the lines of PAPS. 


A short introduction to PAPS is given below, and the PAPS phases are described in Section 3.2. 
The following sections describe the method and application of life cycle costing techniques that are 
appropriate to each of the PAPS phases. Note that Section 3.6 is devoted to programmes where a system is 
bought rather than designed and built. 


For consistency, each section covers: 

e Asummary definition of the phase as given in PAPS. 

e The life cycle costing inputs and outputs expected from that particular phase. 

e An illustration of the benefits that life cycle costing studies can bring in this phase. 

e The types of life cycle cost analysis that can be conducted in this phase. 

e The life cycle costing methods that can be employed in this phase. 

e An example to illustrate the use of life cycle costing in this phase. 

e The process to be followed for conducting life cycle costing in this phase. 

e A method for assessing risk at this phase. 

¢ References where further information can be obtained. 
The PAPS life cycle phase definitions are used in this report because they constitute an internationally 
approved standard. National and other standards differ from these, however. In order to find the section(s) 


in this chapter relevant to a given phase in a project or programme using different terms and definitions, 
it is therefore necessary for the reader to first “translate” these to the equivalent PAPS phases. 


For the nations contributing to this study, the relationship between national standards and PAPS are 
already provided in the information matrices shown in Annex A. Furthermore, an international standard 
for dividing the life cycle into stages exists, defined in the international standard for system life cycle 
processes, ISO 15288. A short description of this, including a description of the relationship between the 
PAPS life cycle phases and the ISO 15288 life cycle stages, is provided in Section 3.10 of this chapter. 
These may also serve as inspiration to readers using other standards. 


3.1.1 Make or Buy 


The PAPS covers the life cycle of a programme from identification of a need for a new capability through 
specifications, design, production, use and eventual retirement of the system. It is general enough to cover 
most national and other life cycle models for this type of programme. However, in many defence 
organisations, particularly in smaller nations, the typical life cycle of weapons systems and other materiel 
systems is substantially different, because systems are bought rather than designed and built. This has 
wide implications for the whole life cycle management process, including life cycle costing, especially in 
some of the earlier phases of a programme. 
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In essence, buying a system involves a choice between relatively few, clearly defined and well described 
and documented alternatives, informed through a process of market research. In contrast, designing and 
building a system from scratch means choosing a solution from a practically infinite number of 
possibilities through a process of design, development, and manufacture. These differences have important 
implications for life cycle costing with regard to the process and method to be followed, the data available 
and the desired results. 


Hence, the difference of a “buy” programme from a “make” programme as described in PAPS is so great 
that it warrants a special treatment. This is given in Sub-section 3.2.9 which covers the decision between 
building or buying a system, and introduces an alternative “procurement” phase to replace the “design 
& development and production phases” in the case where the decision has been made to buy a system. 
Section 3.6 covers life cycle costing in this alternative “procurement phase”. 


3.1.2 PAPS Background 


The handbook on PAPS was published in February 1989 as AAP-20 (Allied Administrative Publication) 
by Defence Support Division of NATO International Staff. 


The stated overall objective of PAPS is “to provide a systematic and coherent, yet flexible, framework for 
promoting co-operative programmes on the basis of harmonised military requirements”. It is further 
stated: “The philosophy behind the PAPS concept is straight forward”. 


There is a finite and fairly consistent number of points (milestones) in the life of a weapon system 
programme when the nature of the programme changes. At these milestones, decisions must be made 
regarding alternative courses of action. PAPS is intended to provide a structured approach to aid decision- 
making at these milestones for all management levels involved in co-operative Research & Development 
and production programmes within NATO. 


PAPS is primarily intended for use in a multi-national weapon system procurement programme in a 
NATO framework, involving two or more nations working together to fill a common capability gap. 
However, the principles laid down in PAPS are applicable to national programmes as well, especially 
since it is repeatedly emphasised that PAPS should in any case be adapted to the conditions of the 
individual project. 


Though the main focus is on the milestones, PAPS also describes the phases between the milestones, 
thus dividing the life cycle of a system into separate well defined phases. An overview of these phases and 
the milestones that separate them is provided in Figure 3-1, short descriptions of the phases are provided 
in Section 3.2, while descriptions of the milestone definitions are found in Annex E. 
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Figure 3-1: PAPS Life Cycle Milestones and Phases. 


3.2 PAPS PHASES 
The following descriptions of the life cycle phases have been drawn directly from AAP-20. 


3.2.1 Mission Need Evaluation Phase 


A statement based on a mission analysis, identifying in broad outline a quantitative or qualitative operational 
deficiency that cannot be solved satisfactorily with existing or planned forces and/or equipment. 


3.2.2 Pre-Feasibility Phase 


The work in this phase will determine whether or not the Outline NATO Staff Target merits a deeper 
feasibility study. It is conducted either by industry and/or government agencies, or by NIAG (NATO 
Industrial Advisory Group). Its aim is to examine the proposal, assess the trade-off points and make a 
broad assessment of the practicable alternatives and also the penalties involved in adopting certain courses 
of action. This study should establish the feasibility of suitable solutions consistent with the timetable of 
needs. The pre-feasibility study will result in the establishment of a NATO Staff Target. 
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3.2.3 Feasibility Phase 


A feasibility study is carried out by industry or government agencies or a combination of both with the 
object of providing a technical appraisal of the feasibility of developing and producing an equipment with 
the performance required by the NATO Staff Target. The study identifies areas of technical risk, 
recommends characteristics of the system(s) and gives the optimum balance between performance, 
cost and development time. The study also indicates areas where considerable advances on the existing 
state of knowledge are likely to prove necessary for successful development. It also indicates the means by 
which the recommended solution will be achieved, suggests a programme for project definition, 
development and production, with a preliminary estimate of the costs for these stages. The feasibility 
study must result in the establishment of a NATO Staff Requirement. 


3.2.4 Project Definition Phase 


This is the process of thoroughly exploring all aspects of the proposed project and to examine the 
relationships between the required performance, development time and cost. The areas of technical 
uncertainty are examined and possible trade-offs are evolved in order to achieve a satisfactory balance 
between performance, development time and cost. These trade-offs may lead to amending the operational 
requirement. From then on, performance requirements and detailed requirements regarding the technical 
characteristics are established so as to meet the operational requirement under the best conditions. 


These requirements will form the basis of the establishment of a development programme and of more 
detailed and realistic estimates of development time and cost. The overall results of the studies carried out 
during project definition will be used for the discussion on whether to proceed with the development or 
not. 


3.2.5 Design and Development Phase 


This phase of a programme calls for design engineering work aimed at full validation of the technical 
approach and ensures complete system integration to the point where production contract action can be 
taken. 


The design and development phase embraces all activities from the preparation of the development 
contract to the approval of the equipment as ready for introduction into service. During the course of this 
phase, the configuration of the equipment is gradually improved. Factory trials are carried out to evaluate 
the results of the design and development activities as far as technology and economics are concerned. 


The engineering work comprises prototype production and technical evaluation trials to determine the 
technical capability of the complete system. The subsequent user trial is designed to test the material under 
realistic conditions. The criteria are tactical mission, military requirements and easy maintenance and 
repair. 


At the end of the phase, the design status for the manufacture of the equipment is determined on the basis 
of statements about technical readiness and field operability. 


A unanimous agreement is required in the steering committee on all provisions, especially operational 
characteristics, financial commitments, agencies involved, follow-on measures, and industrial involvement. 


3.2.6 Production Phase 


The production phase has been defined as: the manufacture of a system, sub-system or equipment in a 
plant or factory using series (i.e. full-scale) manufacturing techniques. 
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The production phase embraces all measures taken to initiate and carry out series production of equipment 
in accordance with the operational requirements and the final development specification up to delivery to 
depots or unit stores. At the beginning of the phase a production contract will be concluded by the 
management office, established by the NPSC (NATO Project Steering Committee). 


By this stage, a statement should be obtained from the Steering Committee on the logistics support and 
training arrangements for the equipment. Whenever possible and appropriate, common logistics support 
(including the option of the NAMSA (NATO Maintenance and Supply Agency) and/or training 
arrangements should be established. 


During this phase several activities aiming at full logistics supportability have to be pursued by the 
management office, such as: 

¢ Production Control. 

¢ Quality Control. 

¢ Acceptance Trial. 

* Codification. 

¢ Configuration Control. 


¢ Modification Trials. 


3.2.7. In-Service Phase 


This is defined as the operational utilisation of equipment by nations. The in-service phase embraces the 
totality of activities aimed at maintaining or restoring full serviceability of the equipment. It includes 
procedures and trials concerning introduction of modifications. 


3.2.8 Disengagement Phase 


This phase encompasses the withdrawal of equipment from operational utilisation in accordance with the 
NADI (National Disengagement Intention). 


3.2.9 Alternative Procurement Phase (Make or Buy Decision) 


The NATO PAPS phases describe the process to be followed in developing and procuring a new system in 
order to fulfil a capability gap. However, at some point in the NATO life cycle the decision has to be made 
either to develop a new system or to procure an off-the-shelf new system. In practice, this would be one of 
the options to be considered. Figure 3-2 shows that in the case of a traditional development programme 
(make track) all the NATO life cycle phases can be followed. Alternatively, in the case of acquiring a new 
system (buy track) some phases from the NATO life cycle will be replaced by a procurement phase to 
allow for a direct acquisition. The figure provides a simplified overview of the PAPS NATO life cycle 
phases for a make or a buy decision. It is always possible to go to a procurement phase from any phase in 
the PAPS process. However, the most suitable moment to decide on a buy decision would be after the 
project definition phase. 
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Figure 3-2: Make or Buy Track PAPS Life Cycle. 


A procurement phase may replace the design & development and the production phase. A more detailed 
description of this phase can be found in Section 3.6. 


3.3. MISSION NEED AND PRE-FEASIBILITY PHASE 


3.3.1 Summary Definition of Phase 


This phase is to determine the operational capabilities of military forces that are required to carry out 
assigned missions, roles and tasks. A comparative assessment is conducted to identify any deficiencies 
that may be related to the element of risk required. This process is then followed by a pre-feasibility study 
to examine the practicable alternatives and establish the feasibility of suitable solutions consistent with the 
time-table of needs. 


The pre-feasibility phase is arguably the most important phase in a programme. It is where decisions are 
made when the amount of supporting information is at a minimum. 


3.3.2 Inputs 


Equipment programmes come into existence as a result of work to identify capability gaps’. These studies 
address the extent of the capability gap, the numbers of equipment or types of platforms required and the 
technologies that can help to fill the gap. A capability gap will trigger the requirement to conduct a 
balance of investment to consider a ‘strategic fit’ in a wider context. Several options will be included here 
including a ‘Do Nothing’ or ‘Do Minimum’ options on legacy systems as well as conducting a thorough 
evaluation on the benefits of procuring new and novel systems. At this phase in the life cycle it is unlikely 
that the costs can be identified in a great deal of detail, rather an understanding of the total programme 
costs and the uncertainty surrounding these estimates is required. 


Since cost and performance data are likely to be immature care should be taken to avoid new conceptual 
proposals being given unwarranted advantage in comparison with those that have been more thoroughly 
explored. For this reason, the processes employed to support and undertake the balance of investment 
normally embrace the following: 


' NATO Capability Based Planning: SAS-057, SAS-063 and AC/327 SG b WG 1. 
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¢ Qualitative approaches that exploit the judgement of military and technology subject matter 
experts who will draw on operational evidence and technology application opportunities. 


¢ Quantitative approaches that will employ mathematical modelling of physical system behaviour 
(principal measurable attributes) within the context of representative operational or business 
situations. 


To support the activities above, a systematic, rigorous and auditable process needs to be adopted. 
This process will be supported by the Outline NATO Staff Target document and an outline CONOPS 
(Concept of Operation) statement from the NATO project steering committee. Cost models that provide a 
holistic (e.g. whole) estimate of cost and time are essential for this phase. The cost models should also 
provide the estimates with defined confidence levels and have the ability to provide a ‘what if’ capability. 


3.3.3. Outputs 


The outputs from this phase should indicate which of the options are feasible and affordable (see also 
Section 2.9) and should be taken forward for further study. The life cycle cost estimates should support the 
refinement of the NATO Staff Target. 


3.3.4 Life Cycle Cost Benefits 


Life cycle costing at this phase is conducted at a very high level and is used to demonstrate the 
relationships between performance (e.g. fleet mix/sizes), procurement policy and life cycle costs. This will 
ensure that all the issues associated with the alternative solutions are considered and evaluated on a 
through life basis. 


3.3.5 Types of Life Cycle Cost Studies 


The types of costing studies being conducted at this phase are predominantly restricted to high-level 
balance of investment, cost benefit analysis and cost-effectiveness studies utilising the life cycle cost as 
one of the measures. Typical examples of these applications are given at Sub-section 3.3.9. All these 
studies are usually supported by Operational Analysis to establish needs and numbers and performance 
measurement criteria. 


3.3.6 Methods Employed 


The life cycle cost estimates derived during this phase usually employ Bayesian, Parametric, Analogous, 
Expert Opinion, or Rule of Thumb techniques. A description of these methods is given at Chapter 4. 
These methods are well established and can provide a holistic estimate to meet the requirements of the 
study. However, care should be exercised when considering an analogous approach. To avoid error, it is 
essential that all comparison using analogy is conducted on a ‘like for like’ basis. Therefore some 
normalisation of the source platform/system/equipment is likely to be required. As in this phase only 
limited quantitative information will be available, also decision support methods or simulation methods 
like system dynamics can be used. 


3.3.7. Life Cycle Costing Process 


Figure 3-3 provides a simple illustration of the life cycle costing process. It shows that at this PAPS phase 
the source data to support the life cycle costing is likely to be immature. Therefore a greater reliance on 
the types of data sources indicated should be expected. In addition, there will be a high level of 
assumptions in terms of the likely performance/design parameters of the systems being evaluated. Risk is 
likely to be measured in a more qualitative rather than quantitative manner. 
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Figure 3-3: Simplified Pre-Feasibility Phase Life Cycle Costing Process. 


It should be expected therefore that the life cycle cost output will be an understanding of the total 
programme costs and the level of uncertainty surrounding them. In summary, the key considerations to the 
life cycle costing process for this phase are: 


e The principal data sources are likely to be parametric based and for speed and ease of use a 
commercially available cost estimating model is likely to be employed. 


¢ There will be many ground rules and assumptions to be recorded and assessed. 


e The risk will be measured at the very top level probably utilising some high level risk register or 
optimism bias technique (a detailed description on risk analysis is given at Chapter 7). 


e The life cycle costs are likely to be reported at the principal cost element level of the generic 
NATO CBS only as shown in Figure 10-1, but probably analysed at a slightly lower level of 
granularity. 


3.3.8 Risk Assessment 


During this phase the identification of risk will be conducted at a very high level. It is likely to be a 
combination of single line statements and will probably contain a mixture of issues as well as risks. 
The cost analyst will need to distinguish the difference between them in order to ensure that only the 
relevant applicable risks are to be included in the cost estimate. 


Where there is no risk register or risk record, an optimism bias technique can be employed. Here it could 
be used to redress over optimistic tendencies by making empirically based adjustments to the cost 


estimates (this technique is explained at Appendix A to Chapter 7). 


At the very minimum, the life cycle cost estimate produced at this phase should include or indicate the 
level of financial risk exposure. 
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3.3.9 Examples 


The following sub-sections demonstrate three alternative applications that are traditionally employed at 
this phase. These are: Balance of Investment, Cost Benefit Analysis and Cost-Effectiveness assessment. 


All the cost estimates for the examples shown were conducted using a Bayesian estimating approach. 
This approach was adopted because: 


¢ No suitable analogies were available without substantial data normalisation thus introducing a 
large uncertainty into the source data to be used for generating the CERs. 


¢ No design data was available to establish nominal or realistic historical based CERs. 


The utility of this approach may be illustrated further by considering the evolution of a programme. At its 
earliest stages, prior to any design or development work, estimates of design characteristics supplied by 
the estimator cannot be anything but imprecise. Through the Bayesian approach, the technique will then 
rely upon the (more certain) design norms, which it generates. As design and development proceed more 
certain information will become available to the estimator for input and estimates will be based 
progressively more upon such data. When design and development are complete design characteristics 
will be known exactly. Further details and an example on the technique are given at Chapter 4. 


To complete the studies the life cycle cost estimates were linked to a decision support tool to provide a 
two dimensional view of the options being evaluated. 


3.3.9.1 Balance of Investment 


Balance of Investment studies traditionally employ portfolio analysis techniques and are used extensively 
in industry and commerce to analyse component business units of a particular portfolio of activities. 
The approach uses a technique, here called Factor Weighting Analysis (FWA), in order to allow the user 
to compare the key characteristics of each activity in comparison with other activities at the same level. 


Portfolio Analysis has been very widely used in North America and Europe particularly in the private 
sector. Considerable research has been carried out to determine how it should be best employed and what 
benefits it confers to the user. Generally speaking, it is now an established part of strategic planning and 
management in the pharmaceutical and other high research industries. This technique is now being tested 
and used to evaluate and support investment strategies for defence portfolios. 


Figure 3-4 shows an example of the results of a portfolio analysis. The diagram shows the estimated 
military value and risk of a group of weapon systems designed to counter enemy sea mines. These are all 
systems in the “detect, classify, and identify” sea mines category or portfolio. Bubbles are sized according 
to resources expended over a 16 year period. Resources in this case include the acquisition costs over a 
six-year period plus ten years of operating and support costs. Risk is an amalgam of assessed values of the 
challenge in developing, transitioning, and operating a system. Finally, the numbers associated with each 
bubble are return on investments or, roughly, expected military value divided by cost, or bubble size. 
Further details on the portfolio analysis technique can be found at Chapter 9. 
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Figure 3-4: Military Balance of Investment Example. 


3.3.9.2 Cost Benefit Analysis 


Cost Benefit Analysis (CBA) is widely accepted as a vital support tool for economic analysis on defence 
programmes. It is primarily used: 


e To assist decision makers. 

¢ To provide transparency of issues. 

¢ To quantify the effect of changes. 

¢ To help in achieving national and multi-national consensus. 


¢ To help to prioritise and compare projects. 


A formal definition of an ‘ideal’ CBA might be: 


“An objective study in which the costs and the benefits of a particular project’s options are fully 
quantified in economic terms, taking full account of the times at which the costs are paid and at 
which the benefits accrue”. 


Usually a CBA is conducted as a net present value analysis (see Sub-section 8.2.2.) by cumulating and 
discounting annual cash flows associated with a particular programme. On the basis of such an approach, 
summary statistics such as net present value, benefit/cost ratio, pay-off periods and internal rate of return 
can be determined. 


In practice, things are not usually quite so straightforward. While costs and most types of benefits can 


generally be quantified after a little research, an economic breakdown of all the projected benefits can 
sometimes be more elusive. 
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The example shown at Figure 3-5 explores the benefits trade-off between live flying, synthetic training 
environments and aircraft in-service life in the provision of current and future heavy lift helicopter and 
light utility helicopter, rotary wing capabilities. 
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Figure 3-5: Benefits Analysis Example. 


The figure presents a top level view of the influences affecting the whole life cost estimate. The items in 
bold text indicate calculation steps (and further influence diagrams) leading to the whole life cost output. 
Input data and intermediate data values are shown as regular text items. 


In this instance the fleet size calculator, the functional analysis framework and the training system cost 
provide the study boundary and the mix of systems to be evaluated. The whole life cost estimates allowed 
the following analysis to be demonstrated: 


¢ The most effective mix between the various aircraft and simulators. 
e¢ When to replace aircraft with simulators. 
¢ The overall value of the return on investment. 


e The timeframe of when the return can be expected. 


3.3.9.3 Cost-Effectiveness Assessment 


In the UK, this is typically referred to as a COEIA (Combined Operational Effectiveness and Investment 
Appraisal). Other nations may use different nomenclature, but in reality, it is the same type of analysis. 
The value of effectiveness is obtained by determining the principal attributes for MOPs (Measures of 
Performance) and MOEs (Measures of Effectiveness) and converting these to a single FOM (Figure of 
Merit). These provide the plots for the vertical axis. The life cycle costs (either constant or discounted) 
provide the measure for the horizontal axis. 
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Figure 3-6 illustrates a typical plot comparing performance and cost. The single value derived plots have 
been adjusted to allow for the uncertainty surrounding the weighting, scoring and cost. From these results 
the decision makers can choose which to take forward for further consideration. 
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Figure 3-6: Cost-Performance Plot Example. 


In this instance, Option 1 can be discounted as Option 2 provides better performance for the same cost. 
Option 4 can also be discounted as it provides the same level of performance as Option 2, but at larger 
cost. Options 5, 6 and 7 are significantly more expensive, but offer benefits in enhanced performance. 
Using this technique, the decision maker therefore has the means to be able to balance the level of 
acceptable performance with the cost available. 


3.3.10 References 


[1] UK MoD, General Instructions and Guidance on IAB and Delegated Approvals, Edition 8, March 
2003, Annex F [unpublished MoD Documents]. 


[2] Decision Analysis for the Professional, Peter McNamee and John Celona (3rd Edition) — SmartOrg. 


[3] Cost-Benefit Analysis: Concepts and Practice (2nd Edition) Anthony E. Boardman, 
David H. Greenberg, Aidan R. Vining, David L. Weimer, Prentice Hall (2000). 


[4] Probability Theory: The Logic of Science, E.T. Jaynes, G. Larry Bretthorst, Cambridge University 
Press. 


[5] Pidd, M. (2004), ‘Systems Modelling — Theory and Practice’, The Management School, Lancaster 
University, John Wiley & Sons Ltd. 


[6] Warren, K. (2004), ‘Competitive Strategy Dynamics’, London Business School, John Wiley & Sons 
Ltd. 
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[7] Coyle, R.G. (2004), ‘Practical Strategy — Structured Tools and Techniques’, Bath University, 
Prentice Hall. 


[8] Coyle, R.G. (1989), ‘A Mission-orientated Approach to Defence Planning’, Defence Planning, 
Vol. 5, No.4. pp. 353-367. 


[9] British Standard EN 60300-3-3:2004, Dependability Management Application Guide — Life Cycle 
Costing. 


[10] HM Treasury Supplementary Green Book Guidance (Joint Service Publication 507). 


[11] UK MoD, Operational Analysis — Foundation for the Business Case, November 2003, Director General 
(Scrutiny and Analysis) [unpublished MoD Documents]. 


[12] www.ams.mod.uk 


3.4 FEASIBILITY AND PROJECT DEFINITION PHASE 


3.4.1 Summary Definition of Phase 


This is where a broad outline of the function and desired performance of new equipment or weapon 
system is assessed. These phases will identify ‘candidate’ system options and solutions and identifies 
areas of technical risk. All the studies conducted during these phases will give an optimum balance 
between performance, cost and time. This will result in the provision of a preliminary estimate for the life 
cycle cost of the overall programme. 


3.4.2 Inputs 


These phases will commence with an agreed and fully documented NST (NATO Staff Target). During 
these two phases, the project changes from a theoretical state to a much more concrete one. The objective 
of the phases is to produce the full definition of the future product from its specifications (e.g. a precise 
description of what it will be from what it is meant to do). 


These are the most important phases from a cost forecasting point of view since many of the decisions will 
have a profound and lasting effect on the project and on its future costs. In fact, so many aspects of the 
eventual system and programme requirements will be frozen during these phases that most of the cost 
reduction possibilities are forfeited during these phases. It is therefore extremely important that a high 
quality life cycle cost estimate is constructed and updated during the whole duration of these phases and 
that it must be used to support the various decision making processes. Typically, the following decisions 
are taken during these phases: 


e The type of technology and material required to build the system. 


¢ The type of contract and industrial structure (including future production rates, support modes and 
all smart acquisition related issues). 


e The possible use of intermediate system introduction and incremental development. 


¢ A strategy of how the system will be used and supported (e.g. ILS studies, etc.). 


3.4.3 Outputs 


The outputs from the feasibility phase will be a detailed NSR (NATO Staff Requirement) and, from the 
project definition phase, a NADDO (NATO Design and Development Objective). To complete these 
documents the life cycle cost estimates will be supported by industry data. Some design data will be 
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available. The life cycle cost estimates will be available at all the major line item level in the cost 
breakdown structure. 


3.4.4 Life Cycle Cost Benefits 


Life cycle costing is conducted at a much more detailed level than the previous phases. More information 
should be available on the system design and logistic support. This will make the life cycle cost and 
logistic support analysis much more substantive and meaningful. 


3.4.5 Types of Life Cycle Cost Studies 


The type of life cycle cost studies to be conducted during this phase varies according to the degree of 
precision required by the decision to be taken. In general, the costs become more precise and dedicated to 
some aspects of the programme as it progress in time. Typical examples of studies include: 


¢ The evaluation of technological choices requiring an analytical description of the product, and the 
scope of the cost estimate may need to consider various sub-assemblies or even components 
within the whole system. 


¢ Major procurement issues (depending on the proposed acquisition strategy) may require a much 
broader approach and necessitate expanding the costing boundary to include elements 
(e.g. facilities and personnel costs) often only considered at later phases in the programme. 


When comparing different solutions, it is essential to verify that the costing boundaries (scope) considered 
in the life cycle cost estimates are consistent and that the capabilities provided are comparable. Particular 
note should be taken when no alternative solution is apparent. This would result in a decision to: 


¢ Not to acquire the capability (which may have a political cost) or, 


¢ To keep using existing systems (with, usually, increased in-service costs due to the need to 
overcome obsolescence or recover natural wear and tear). 


In conducting life cycle cost estimates on multi-national programmes, great care must be applied to the 
assumptions of deployment. Although each nation may have slightly different deployment and 
supportability, for cost modelling purposes, a consensus should be agreed so that there is consistency in 
the life cycle cost outputs. In this way, optimised solutions can be evaluated to address both national and 
multi-national concerns. 


3.4.6 Methods Employed 


During the previous phases the life cycle costs would have been derived using a form of parametric 
approach with many assumptions. The risk assessment would have been conducted at a high level and, as 
expected, the overall level of uncertainty would have been significant. 


During the feasibility and project definition phase the level of design detail available will be increased 
thus allowing for refinement of any parametric based estimate. This will be supplemented by information 
(e.g. system breakdown, component reliability, system maintainability, etc.) from industry and further 
information on the planned logistic support. This will allow a more detailed life cycle costing to take place 
and probably also using some optimization methods. Although in this phase more detailed information 
will become available, the method of analogy will still be used quite often employing in-house cost 
models. Also at this phase a detailed and fully quantified risk register should be available for conducting a 
cost risk analysis. 
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3.4.7 Life Cycle Costing Process 


Figure 3-7 provides a simple illustration of the life cycle costing process. It shows that at this PAPS phase 
the source data to support the life cycle costing is likely to be more mature in terms of system engineering 
design therefore the total reliance on parametric approaches will be reduced, but the refined parameters 
should still be used as a sanity cross check. More data should now be available from industry and a more 
detailed population of the cost breakdown structure can commence. It should be possible to populate all 
the major line items in the cost breakdown structure. 
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Figure 3-7: Simplified Feasibility and Project Definition Life Cycle Costing Process. 


All the risks should now be captured and quantified within a risk register to provide the basis for 
conducting the cost risk analysis. 


At the completion of this phase it should be expected that the life cycle cost output will be an 
understanding of the total programme costs (all major elements of the cost breakdown structure) and the 
level of uncertainty surrounding them. In summary, the key considerations to the life cycle costing process 
for this phase are: 


e The principal data sources are likely to be parametric based and likely to be supplemented by 
information from industry. 


¢ Some design data is likely to be available to refine the estimates and provide an initial baseline for 
the operating and support costs. 


¢ The life cycle costing is likely to be conducted using in-house models and the results cross- 
checked and supplemented (if possible) using an alternative method. This provides two 
independently developed estimates to support the robustness of the life cycle cost estimate. 


¢ There will be many ground rules and assumptions to be recorded and assessed. 


e The risk will be measured at a reasonable level of detail probably utilising a quantified detailed 
risk register and cross-checked using an optimism bias technique. 
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e The life cycle costs are likely to be reported at the principal cost element level of the generic 
NATO CBS only as shown in Figure 10-1, but probably analysed at a slightly lower level of 
granularity. 


3.4.8 Risk Assessment 


In the feasibility and project definition phase a more detailed risk analysis will be conducted than in the 
earlier phase. The risk will be measured at as detailed a level as possible probably using a quantified risk 
register. If possible an optimism bias technique should be used. See for an overview of the optimism bias 
technique Appendix A to Chapter 7. 


3.4.9 Example 


As previously mentioned, this phase requires close, continuous collaboration between the research 
community, the user, and the developer as alternative technologies are assessed and as performance 
parameters are refined. As Figure 3-8 shows, life cycle cost estimates of alternative options are critical 
inputs in choosing the best future system or technology (Path 1 or Path 2, in this simple example) 
for meeting emerging war-fighter requirements. 
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Time —————————__> 


Figure 3-8: End-State Solution. 


Given that several, different, potential technological solutions are typically examined, and given that user 
requirements are still very fluid, the method of choice for performing life cycle cost estimates in this phase 
is usually parametric. This allows for rapid analytical response to changes in technologies and user needs. 


To take a concrete yet simple example (see Figure 3-9), assume a new satellite needs to be developed to 
detect stealthy enemy sea mines in very shallow water (VSW), in the littoral. The user community is 
unsure of their specific requirement concerning the size of enemy sea mines that it needs to detect. 
The range of user values is from one to three cubic metres. This requirement, in turn, drives the size and 
power of the sensor and, consequently, the size of the satellite bus and the number of thousands of sources 
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lines of code (KSLOC) to develop. Note that detecting a relatively small enemy sea mine requires a 
relatively powerful sensor and hence more lines of code and a bigger bus (to carry a bigger sensor). 


| CER Input Variables fee A be Be ee) 


Sensor Fidelity (size of sea mine) | 1 cubic metre | 2 cubic metres | 3 cubic metres 


Software (KSLOC) 250 
Bus Weight (pounds) 1500 


Figure 3-9: Example Sensor Fidelity to Software Size. 


Three CERs are use to generate a cost estimate for development for each of the three cost elements: 
¢ Payload (sensor) 
e« Expected Cost = 3,568,510*(size of sea mine in cubic meters) ”*’ 
¢ Software 
* Expected Cost = 435,216*(KSLOC)°*”! 
¢ Bus 
* Expected Cost = 87,450*(Bus Weight)”” 
Similarly, cost estimates for production and for operating and support are generated using parametric 


techniques. Finally, costs for each phase are rolled-up to produce life-cycle cost estimates for the three 
alternatives. 


3.4.10 References 


[13] Handbook on the Phased Armaments Planning System (PAPS), Allied Administrative Publication 
(AAP-20) PAPS, NATO Defence Support Division, 1989. 


[14] UK MoD, Operational Analysis — Foundation for the Business Case, November 2003, Director 
General (Scrutiny and Analysis) [unpublished MoD Documents/Reports]. 


[15] www.ams.mod.uk 


3.5 DESIGN AND DEVELOPMENT PHASE 


3.5.1 Summary Definition of Phase 


This phase of the programme calls for sufficient design engineering work to be conducted to ensure that 
full validation of the technical approach can be achieved. Also, the complete system integration is assessed 
to the point where production contract action can be taken. 


3.5.2 Inputs 


This phase commences with the NADDO (NATO Design and Development Objective) and is the phase 
where the earnest work on life cycle costing is performed. In this phase the design and development of the 
system being considered is well under way. An equipment, a system or a platform has been chosen and the 
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next step is to develop a final design and to choose a supplier. A lot of information from earlier phases 
may exist; in-house historical data, results from early tests and technical demonstration. 


Ideally in this phase, a RFI (Request for Information) can be initiated and discussions can be held with 
several potential suppliers without commitment. This can provide several advantages. Information 
received from these contacts can provide a better understanding of the programme and introduce the 
opportunity to influence the life cycle costs by using market competition. The process also provides an 
opportunity to further develop the cost breakdown structure (greater transparency and higher precision). 
This approach to gaining data and information is also applicable in situations where only a single supplier 
is available. 


In addition, the CONOPS (Concept of Operations) document should be used in conjunction with the ARM 
(Availability, Reliability and Maintainability) information provided by the supplier to refine the operating 
and support costs. 


3.5.3. Outputs 


On completion of this phase the life cycle costing will be sufficiently comprehensive and complete to 
support the NAPO (NATO Production Objective) and provide a realistic forecast of the likely total whole 
life cost. 


3.5.4 Life Cycle Cost Benefits 
The main purposes of life cycle costing in this phase are to: 


e Influence the alternative solutions by evaluating the balance between life cycle costs and military 
capability. 
e Support the system choice through using the results of the life cycle costing. 


e Establish the logistic support requirements for the chosen solution to minimise the life cycle costs. 


It is an established fact that many of the cost drivers are influenced by design decisions. The challenge for 
the system engineers is to balance the acquisition and support cost to provide the minimum life cycle cost. 
The life cycle costing activity plays a very significant role by providing data for systems engineering trade 
studies aimed at minimising the life cycle cost. 


3.5.5 Types of Life Cycle Cost Studies 


Entering the design and development phase concludes a decision to whether to make or buy a system off- 
the-shelf to meet the requirement capability. Since the design and development phase is a part of the 
“make track”, there is a relative large extent of freedom in designing the end product. 


The types of life cycle cost studies to be employed can be described in many dimensions. A split can be 
made between: 


e A life cycle cost analysis that aims toward costing the production and operation of the system(s), 
and 


¢ Costing the alternative effects of introducing the system into the organisation in addition to or 
incrementally replacing the first. 


Experience shows that up to 80 % of the life cycle costs relating to the system can be influenced in the 
design phase. This demonstrates that significant resources should be allocated to the life cycle costing 
work to be conducted during this phase. 
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The life cycle costing studies should be performed as an iterative process with each cycle providing 
greater levels of completeness, granularity and clarification. Any changes to the design should be aimed at 
minimising the life cycle costs whilst meeting all the desired requirements. The life cycle costing work 
should focus on both the principal system and all supporting infrastructure. 


Cost savings are often found during this iterative cost estimating process. It is therefore essential that a 
thorough analysis of all the cost elements is conducted at this phase. 


3.5.6 Methods Employed 


There are no limitations to what type of methods can be employed in this phase. For example, engineering, 
parametric, and analogy are all still applicable and can be used to produce a life cycle cost analysis of high 
quality. It is often found that a combination of these methods is useful particularly when used together 
with expert opinion. 


However, in order to provide a high quality input to these methods, a close working relationship will need 
to be maintained with the system design team. 


3.5.7 Life Cycle Costing Process 


Previous life cycle costing studies will have been conducted by government departments or their agents 
using a variety of methods, various assumptions and predictive system design data. At this phase it is 
appropriate to firm up as much as possible on the assumptions and predictions by gaining further 
information from industry. The major steps to developing the life cycle costing further during this phase 
are: 


e« Issue a RFI (Request for Information), based on the cost breakdown structure, to several 
prospective suppliers requesting all the data to support a life cycle cost study of their proposed 
system. 


¢ Conduct a life cycle cost study to identify the key cost drivers, the costs that can be influenced 
and the costs that differ between the alternative solutions. 


¢ Validate the system specification based on a cost/benefit analysis and the associated risk and 
uncertainty. 


e Evaluate the life cycle cost analysis to support the procurement decision. 


e  Iterate the process to resolve any conflicts between system performance and cost. 


Each of these 5 steps is described in more detail below. 


3.5.7.1. RFI and CBS from Vendors — LCC Comparison 


During this phase, it may be possible to have a dialogue with several prospective suppliers in order to gain 
information to support the life cycle costing process. This can provide a better understanding of their 
systems and increase the possibility to influence the life cycle costs by making the cost breakdown data 
more transparent and with greater precision. This approach is also applicable in situations where there is 
only a single supplier. By using this approach, different product designs can be assessed in terms of a 
minimum life cycle cost solution. 


To enable comparisons between different product designs, it is crucial to ensure that the costing boundary 


is consistent. For example, using the same definition for personnel cost in the cost estimates given by the 
suppliers or ensuring whether the personnel cost has been included or not. The Operating and Support cost 
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elements should be determined using an agreed CONOPS (Concept of Operations) to highlight all the 
deployment requirements. 


It is also essential to understand how the life cycle costing has been conducted by the suppliers and the 
models they have used to conduct the analysis. 


A risk register together with a quantitative risk assessment should be provided by each of the prospective 
suppliers. This should be based the suppliers ability to meet the requirements detailed in the NADDO 
(NATO Design and Development Objective). 


All the data and information gathered from the prospective suppliers should be benchmarked against 
empirical data either from internal or external sources. This should diminish the risk of a biased life cycle 
cost analysis. 


3.5.7.2 Cost Drivers — Influencing the LCC 


When receiving the completed cost breakdown structure from the prospective suppliers the focus of the 
life cycle cost analysis should be on identifying the major cost drivers and avoiding becoming embroiled 
with too much detailed data. The focus on major cost drivers is needed in order to reveal potential areas of 
product design changes that could have a significant influence in minimising the life cycle cost at the 
earliest opportunity. 


The focus of the analysis should also be on the cost drivers that enable comparability between the different 
product designs. It is more crucial to reveal the cost differentiation between differing product designs than 
estimating the likely total cost. 


3.5.7.3. Validate the System Specification 


The cost drivers are further analysed to understand the effects on both benefits (capabilities) and the cost 
of any possible changes. This would include requesting the suppliers to clarify data on the identified cost 
drivers. In the original data analysis, risk premiums with regards to new technology should be included 
and the suppliers risk analysis should also be reviewed together. The resulting analysis will support the 
decision making process and could result in adjustments being made to the final specifications in order to 
achieve a lower life cycle cost. 


3.5.7.4 Evaluation and Iteration 


In practice, life cycle costing should be part of the iterative process of the system design. The costing 
boundary defines exactly what elements should be included and the level of detail in which they will be 
considered. The life cycle costing activity during this phase should be sufficiently robust to support a 
‘value for money’ decision. To achieve this level of confidence it may be necessary to iterate the life cycle 
costing process several times during the phase. It is essential to ensure that life cycle costing is not just a 
one-off activity. To be at its most efficient it needs to be integrated into the design process, be able to 
identify opportunities for cost savings and be accurate in terms of providing a robust and defendable life 
cycle cost. 


Figure 3-10 provides a simple illustration of the life cycle costing process. It shows that at this PAPS 
phase the source data to support the life cycle costing is likely to be very mature in terms of system 
engineering design therefore there should be detailed design data (structure and ARM) available from 
industry. Logistic information providing support analysis and stockholdings should also be available. 
All this information will allow a detailed population of the cost breakdown structure. It should now be 
possible to populate all the line items in the cost breakdown structure. 
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Figure 3-10: Simplified Design and Development Phase Life Cycle Costing Process. 


All the risks should now be captured and quantified within a risk register to provide the basis for 
conducting the cost risk analysis. There should also be mitigation plans for all the major risk areas and the 
costs of undertaking these should also be included in the cost estimates. 


At the completion of this phase the life cycle cost output should be a detailed account of all the line items 
contained in the cost breakdown structure. A comprehensive risk analysis in terms of Cost and Schedule 
impact should also be available. 


In summary, the key considerations to the life cycle costing process for this phase are: 


The principal data sources are likely to be design and logistic information from industry 
supplemented by military data on likely deployment and staffing numbers. 


All design data and a comprehensive CONOPS report is likely to be available to refine the 
estimates and provide an established baseline for the acquisition, operating and support costs. 


The life cycle costing is likely to be conducted using in-house models and the results cross- 
checked and supplemented using an alternative method. A further independently generated 
estimate would provide a third view and provide the assurance and robustness of the life cycle 
cost estimate. 


All the ground rules will have been agreed. There will be few assumptions to be recorded and 
assessed. Although, it can be expected that the reliability and maintainability modelling will have 
been conducted using predictive data. The criticality of this data should be measured using 
sensitivity analysis. 


The risk will be measured at a detailed level utilising a quantified detailed risk register and 
recognised risk simulation models. 


The life cycle costs are likely to be determined at all the line items in the cost breakdown 
structure. 
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3.5.8 Risk Assessment 


During these phases the identification of risk should be conducted at a very comprehensive level to fully 
understand the level of risk exposure to cost and time. All the mitigation plans should have been 
developed and programmed. The cost of conducting the risk mitigation actions should be included in the 
life cycle cost estimate. 


The risk analysis should comprise pre and post risk mitigation scenarios as well as developing the time 
line for undertaking and completing the mitigation actions. 


At completion of the phase the results of the risk analysis should provide a clear indication on the level of 
financial and timescale risk exposure to the programme. 


3.5.9 Examples 


A number of examples are shown that relate to the types of studies to be undertaken before and after 
contract award. It is important to distinguish these as the added value provided to this and the next phase is 
invaluable. 


3.5.9.1 Analogy Cost Estimating 


This cost estimating method is accomplished by forecasting the cost of the future based on the historical 
cost of a similar or analogous item. The costs of the historical item must first be normalised for both 
content and historical price differences. Normalising for content entails deducting the cost of components 
that are not comparable to the new design and adding estimated costs of the new components. Normalising 
for inflation entails converting historical cost to an appropriate base year value and applying the proper 
escalation indices to achieve then-year costs. Estimating by analogy involves comparing your system 
and/or cost breakdown structure elements to comparable current and or historical systems or cost 
breakdown elements. This involves understanding the programme and how it derives its history, 
for example, what programme it is based upon. It is important to interact with programme engineers to 
ensure the validity and credibility of candidate analogous programme to the future system; once 
comparable programmes are considered, it is necessary to seek out those specific systems if possible to 
obtain necessary data and cost information. The estimator will also need to talk to the programme 
engineers to understand differences between the future system and the comparable analogous system(s).” 


Figure 3-11 is a simple example of estimation by analogy for military personnel pay systems. System A is 
old and system B is new. The objective is to estimate the software development effort for the new system. 
This is done using system A as an analogy, or more specifically, using man-months of effort per function 
point on A as the analogous ratio or multiplier, to be applied to the new system, B. 


* NAVSEA 2005 Cost Estimating Handbook, p. 4-11. 
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Figure 3-11: Example of Analogy Cost Estimating. 


Finally, complexity factors have often been used to adjust analogous estimates. However, they can often 
undermine the credibility of the future estimate if they have not been substantiated. For example, 
engineers might suggest that a new programme is twice as complex as an analogous programme, and that, 
therefore, the new programme’s cost should be twice the cost of the old programme. In reality, 
the relationship between complexity and cost might be unknown. It could be proportional, linear, 
or exponential. Without hard data, a subjective adjustment will negate the credibility of the estimate. 


3.5.9.2 Parametric Cost Estimating 


Parametric estimating requires that a statistically valid relationship be established among the dependent 
variable, such as cost, and independent variables, such as costs of other elements, and or various physical 
and performance characteristics of that system. This parametric CER (cost estimating relationship) is then 
used to estimate the cost of a new system with different values for the same physical and performance 
characteristics.” 


Provided below is an example, based on a dozen historical observations on number of man-months 
required for lead ship construction and the displacement tonnage of that ship. Note, that construction for 
the first ship in a class is typically a development contract, and includes non-recurring design effort. 


First, a CER is estimated using least-squares regression analysis. 
Y; = 10.3 + 100.7X;+ ej 


(4.3) (3.2) 

n =12 

R? =0.87 

F =17.1, 

where n 7 the number of historical observations used to estimate the equation. 

Y; a number of man-months required to construct the i” ship (in millions). 
X; a displacement tonnage of the i” ship (in thousands). 
ej = the i” regression residual. 


> NAVSEA 2005 Cost Estimating Handbook, p. 4-11. 
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R = The coefficient of determination, or the proportion of variation of Y (dependent 
variable) that can be attributed to the variation of the explanatory or independent 
variables (X’s). It is a measure commonly used to describe how well the sample 
regression line fits the observed data. Note that 0 <= R’ <= 1. 


F = measures the overall power of the regression equation in explaining changes in the 
dependent variable. More specifically, it is used in testing the null hypothesis of no 
relationship between the set of X’s (explanatory variables) and Y (the dependent 
variable). 


and where t-statistics are shown in parentheses. 


Then, a value for displacement tonnage for the new system is input into the estimated regression equation 
to give an estimate of man-months of effort required to design and construct the new ship. 


3.5.9.3. Historical Trend Analysis 


In addition to the estimating techniques discussed above conducting a historical trend analysis provides 
the estimator with the ability to set estimated costs and time schedules either separately or together in the 
context of historical costs for similar or related programmes. This is distinct from any cost estimating 
models themselves. Its uses are: 


¢ To provide supplementary background information of a kind that can assist an estimator by giving 
a wider context to their work upon an individual programme. 


e Similarly, by providing context and background to a study, it facilitates the work of those 
responsible for the scrutiny of estimates prepared by others e.g. as in auditing and due diligence 
work. 


Figure 3-12 illustrates the estimated cost of a new programme in the context of historical trends as 
exemplified by the actual outturn costs of a substantial number of similar programmes completed in the 
past. All the costs having been normalised to common economic conditions and due allowance made for 
differences in scale. How well the estimated costs of the new programme fit into this context is both seen 
by the user in readily-understood graphical form and can also analysed statistically. 
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Figure 3-12: Example of Historical Trend Analysis. 
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3.5.9.4 Example of Evaluation of Contract Performance 


Once a contract has been awarded for design and development, or production, data from an EVM (earned 
value management) system can be employed as an aid to decision making. EVM is an integrated 
management control system for assessing, understanding, and quantifying what a contractor is achieving 
with programme funding. More specifically, EVM: 


e Integrates technical, cost, and schedule information with risk management. 

e Allows objective assessment and quantification of current project performance. 

¢ Helps predict future performance based on trends. 
Earned value provides an objective measurement of how much work has been accomplished on a 
programme. Using the earned value process, the management team can readily compare how much work 
has actually been completed against the amount of work planned to be accomplished. All work is planned, 
budgeted, and scheduled in time-phased ‘planned value’ increments constituting a PMB (Performance 


Measurement Baseline), or BCWS (Budgeted Cost of Work Scheduled) curve, as shown in Figures 3-13 
and 3-14, which have been taken from the US Defense Acquisition University EVM Gold Card. 
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Defense Acquisition University 
Earned Value Management Gold Card 


EAC 
TAB 
Management Reserve BAC 
Completion 
Date 
VARIANCES (Favorable is positive, Unfavorable is negative) 
* Cost Variance Cv = BCWP-ACWP CV%= CV 7 BCWP 
* Schedule Variance SV = BCWP- BCWS SV%= SV/ BCWS 
* Variance at Completion VAC =BAC-EAC 
PERFORMANCE INDICES (Favorable is > 1.0, Unfavorable is < 1.0) 
* Cost Efficiency CPI = BCWP’ ACWP 
* Schedule Efficiency SPI = BCWP /’ So 
OVERALL STATUS 
P c = BCWPCUM 
+* Percent Complete Bac! 
¢ Percent Spent - -ACWPCUM __ 
BAC 
TO COMPLETE PERFORMANCE INDEXCP) 
TCPI - WORK REMAINING - BAC-BCWPCUM 
(EAC) COST REMAINING EAC- ACWPCUM 
ESTIMATE AT COMPLETION (EAC = ACWP+ Estimate for Remaining Work) 
BAC z BAC-BCWPCUM 
EAC = ——— *EAC, ite= ACWPCUM + ee 
or CPICUM aaaae: (CPI CUM) + 6PICUM) 


Figure 3-13: Example EVM Calculations. 
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+ OVERRUN 


Control Accounts Undistributed Budget 


Work Packages Planning Packages 


_TERMINOLOGY 
NCC -—-Negotiated Contract Cost Contract price less profit / fee 
AUW - Authorized Unpriced Work Work authorized to start, not yet negotiated 
CBB -Contract Budget Base Sum of NCC and AUW 
OTB - Over Target Baseline Sum of CBB and recognized overrun 
TAB - Total Allocated Budget Sum of all contract budgets - NCC,CBB or OTB ( includes MR ) 
BAC - Budget At Completion Cumulative BCWS - total end point of PMB ( excludes MR ) 
PMB - Performance Measurement Contract time-phased, budgeted work plan ( excludes MR ) 
Baseline 
MR -—Management Reserve Contractor PM’s Contingency budget 
UB -Undistributed Budget Broadly defined activities not yet distributed to CAs 
CA -Control Account Contractor key management control point - CWBS element 
WP -—Work Package Near-term, detail-planned activities within a CA 
PP -—Planning Package Far-term CA activities not yet defined into detail Work Packages 
BCWS - Budgeted Cost for Work Scheduled Value of work scheduled -- PLAN 
BCWP - Budgeted Cost for Work Performed Value of work completed -- EARNED VALUE 
ACWP - Actual Cost of Work Performed Cost of work completed -- ACTUAL COSTS INCURRED 
EAC - Estimate At Completion Estimate of total contract costs 


EVM POLICY (DOD 5000.2-R) 
ALTERNATIVE EV MANAGEMENT APPLICATIONS 


LEVEL 1. EVMS Industry Standards Management Application 
Contractor management system certified as meeting Industry Standards 
¢ Required for non-FFP contract exceeding $73M RDT&E or $315M in procurement (CY00$). 
- PM may apply to contracts below-threshold —consider benefits, risk and criticality. 
* Contractor must establish, maintain, and use a system that meets the the 32 Industry Standards. 
* Cost Performance Report (CPR) delivered as a CDRL item. 
- 5 Formats (WBS, Organization, Baseline, Staffing, and Explanations) 


LEVEL 2. C/SSR Management Application 
Contractor Management system not certified 


- Required for non-FFP contract exceeding $6.3M (CY00$) and 12 months in length. 
-’Reasonably objective’ EV methods acceptable, traceability at higher level (CA vs WP) 
*The CPR or the Cost/Schedule Status Report (C/SSR) delivered as a CDRL item. 


EVM Home Page — http:/www.acq.osd.mil/pm! 
DSMC EV E-Mail Address — EVM@DSMC.DSM.MIL 


DSMC EV Phone No. — (703) 805-2848/2968 (DSN 655) June 2000 


Figure 3-14: Example EVM Process. 
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As work is performed, it is ‘earned’ on the same basis as it was planned, in monetary or other quantifiable 
units such as labour hours. Comparing earned value with the planned value measures the monetary value 
of work accomplished versus the monetary value of work planned. Any difference is called a schedule 
variance. 


Earned Value — Planned Costs = Schedule Variance (SV) 


The value earned for the work performed compared with the actual cost incurred for the work performed 
(taken directly from the contractor’s accounting systems), provides an objective measure of cost 
efficiency. Any difference is called a cost variance. 


Earned Value — Actual Costs = Cost Variance (CV) 
A negative variance means more money was spent for the work accomplished than was planned. 
Conversely, a positive variance means less money was spent for the work accomplished than was planned 


to be spent. Finally, as the figures indicate, cost and schedule variances can be used to generate estimates 
of contract cost at completion (EACs). 


3.5.10 References 
[16] NAVSEA 2005 Cost Estimating Handbook, http://www.navsea.navy.mil/sea017 


[17] Earned Value Management Implementation Guide, Defense Contract Management Agency, October 
2006; available from http://guidebook.dcma.mil/79/evmigoldversion.doc 


[18] U.S. Defense Acquisition Guidebook available from http://akss.dau.mil/dag 


[19] DoD Instruction 5000.2 “Operation of the Defense Acquisition System”, May 2003 available from 
http://akss.dau.mil/dag 


[20] UK MoD, General Instructions and Guidance on IAB and Delegated Approvals, Edition 8, March 
2003 [unpublished MoD Documents/Reports]. 


3.6 ALTERNATIVE PROCUREMENT PHASE (MAKE OR BUY DECISION) 


3.6.1 Summary Definition of Phase 


As described in Section 3.2.9 this alternative procurement phase replaces a design and development phase 
and the production phase to represent a situation where an off-the-shelf buy (instead of a make) decision is 
made. In this situation, only a limited number of activities can be expected. 


3.6.2 Inputs 


The input for the alternative procurement phase is the agreed and fully documented NATO Staff 
Requirement. However, the level of detail required (in terms of performance and supportability) from 
industry would be far more substantial. This would be gained from formal approaches such as an RFI 
(Request for Information) or a more contractual RFQ (Request for Quotation). If there are more than one 
potential suppliers then a tender competition may be conducted and the ITT (Invitation to Tender) could 
comprise a life cycle cost questionnaire (see Annex C for an example) for completion by the contractors. 
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3.6.3 Outputs 
The life cycle cost outputs should be sufficient to fully understand, on a year by year basis, all the 


acquisition, operating and support costs over the life of the programme. 


The output of the alternative procurement phase will be a product or system that complies with the NATO 
Staff Requirement. 


3.6.4 Life Cycle Cost Benefits 


The main purposes for conducting life cycle costing on an off-the-shelf procurement programme is to 
provide the decision maker with a full understanding of the financial commitment over the life of the 
programme. 


3.6.5 Types of Life Cycle Cost Studies 
The type of life cycle cost studies to be conducted in this phase can be summarised as follows: 
¢ Supporting the tender evaluation process. 


In the tender evaluation process the life cycle costing methodology can be used to ensure that the 
contract is award to the tender offering a system that meets all technical and availability 
requirements at minimum life cycle cost. The cost of investment in maintenance resources and the 
cost of lifetime support will be weighed against the cost of investment in the technical system. 
The resulting life cycle cost will be included in the overall tender evaluation period. 


¢ Comparing alternative solutions or options on costs, in order to choose the best solution. 


As part of the alternative procurement phase various alternatives need to be compared. In some 
cases, the tender evaluation process may have to evaluate several options provided in the tender 
response all of which may meet the requirements (including direct procurement or a lease option). 
Or it may be that a nation would like to compare various types of support: e.g. own maintenance 
organisation versus outsourcing. In this case it is not necessary to estimate all cost elements. 
It may be sufficient only to estimate the cost elements that will show differences between the 
options. These relative comparisons however, will not provide a complete picture of the life cycle 
costs, but would be an aid to the decision making process. 


¢ Determining the total life cycle costs 


In order to make a fair decision in the procurement phase, a nation would need to have an 
overview of the total life cycle costs, not only to determine the forecast of future spending, 
but also to assess the affordability of a procurement programme. In this case, all the cost elements 
have to be estimated and a detailed cost study is required. 


3.6.6 Methods Employed 
Dependent on the data available, many methods may be used in this phase: 
e Engineering method. 
e Parametric method. 
¢« Analogy method. 
e Expert opinion. 
¢ Rules of thumb. 


¢ Catalogue. 


RTO-TR-SAS-054 3-29 


LIFE CYCLE COSTING IN THE PAPS PHASES anime 


For more details on these methods see Chapter 4. 


3.6.7 Life Cycle Costing Process 


The major steps to follow are similar to the process described in Section 3.5.7. However, in an alternative 
procurement phase it is conducted in less detail. 


¢ Based on the system requirements described in the NATO Staff Requirements an RFI/RFQ/ 
RFP/ITT is compiled and sent to several companies and they are asked to answer cost related 
questions and fill in data in an agreed cost breakdown structure format. This will include a 
statement on the definitions and the basis of the cost estimate. 


¢ Gathering data from internal sources in order to complete the cost model. 


¢ Data analysis to find cost drivers, costs that can be influenced and costs that differ between the 
alternative solutions. 


¢ Evaluate the result, either to support the procurement decision or to iterate the process until the 
basis for decision is clearly understood and agreed. 


More details are provided at Sub-section 3.5.7. Figure 3-8 can also be used for the alternative procurement 
phase. 


3.6.8 Risk Assessment 


During the tender evaluation the identification of risk should be conducted at a very comprehensive level 
in order to fully understand the level of risk exposure. All the risk mitigation plans should have been 
developed and programmed and assigned to owners. The cost of conducting the risk mitigation actions 
should also be included in the life cycle cost estimate. 


The risk analysis should comprise pre and post risk mitigation scenarios and should develop the time line 
for undertaking and completing the mitigation actions. At completion, the results of the risk analysis 
should provide a clear indication on the level of financial and timescale risk exposure to the programme 
prior to contract negotiation. 


3.6.9 Example 


The multi-national NSHP (Nordic Standard Helicopter Programme) is an example of how the life cycle 
costs can support the tender evaluation process. 


All the participating nations’ requirements were implemented in a common Nordic Requirements 
Document. This document defined those requirements that were considered common and those which 
were specific to a particular nation. The request for quotation required prospective contractors to provide 
detailed data to enable the evaluation of performance and reliability. In addition, this information would be 
used to estimate the life cycle cost. The evaluation of system availability and life cycle cost were 
performed for each participating nation’s individual life cycle cost model. 


Figure 3-15 represents a summary of user data and tender data that was included in each nation’s life cycle 
cost model. The figure also shows how the calculated estimates for cost of spare parts were used as input 
to the model. All the nations’ specific life cycle cost models were then combined in a total life cycle cost 
model, representing a combined life cycle cost for all the participating nations. 
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USER DATA Lcc TENDER DATA 
¢ Aquisition cost 
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Optimisation Availability Requirement 
“> Max waiting time 


Figure 3-15: LCC Data Process. 


The requirement for spares was calculated, by the procurement agency, using a logistics and spares 
optimisation model, based on the individual nation’s operational profile. The criterion for the cost 
modelling was based on the MWT (Mean Waiting Time) for spare parts together with the unavailable time 
due to preventive and corrective maintenance. The system availability requirement was given as 80% 
(see Figure 3-16). By using this approach all the tenders were normalised and compared using the same 
performance level. The calculated costs for spare parts were used as an input to the life cycle cost 
evaluation. 
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Figure 3-16: Availability Precedence. 


Some of the results from the life cycle cost tender evaluation are presented below. Figure 3-17 shows the 
sum of all the participating nations’ life cycle costs per tender. The cost for total life cycle cost, 
acquisition, operation and life support have been normalised to Tender F to provide a relative comparison. 
The life cycle cost tender evaluation process generated many different diagrams that presented different 
aspects and the cost drivers for each tender. 
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Tender F 100% 62% 15% 23% 
m Tender G 108% 74% 16% 18% 
co Tender H 94% 55% 15% 24% 
co Tender J 131% 72% 16% 43% 


Figure 3-17: LCC Evaluation Results. 
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When analysing the LSC (life support costs) (Figure 3-18) in more detail it became clear that the 
investment in spare parts differed significantly between the tenders. The difference was explained by the 
different maintenance concepts offered by each contractor. 


LSC, All nations's helicopters, 20 years 
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Figure 3-18: LSC Evaluation Results. 


Figure 3-17 shows that Tender H has the lowest costs, however Tender F was selected as the winning 
tender due to other considerations. The life cycle cost analysis results were used as a baseline for 
negotiating the contract. In addition, the life cycle cost analysis is now being used to establish a support 
cost baseline prior to negotiating a contractor logistic support contract. 


3.6.10 References 


[21] Handbok for driftsékerhet (H DriftSak), 2006, (M 7740-714001 H Driftsak), Swedish Armed Forces 
and Swedish Defence Materiel Administration. (in Swedish). 


[22] Stappenplan voor het Analyseren van Levensduurkosten binnen de Defensie Organisatie (SALDO), 
FEL-93-A337, december 1993, B. Bhola and J.A.M. van Wees (in Dutch). 


[23] Aanwijzing DGM inzake levensduurkosten bij materieelprojecten, aanwijzing 98-4 d.d. 11 december 
1998 (in Dutch). 


3.7 PRODUCTION PHASE 


3.7.1 Summary Definition of Phase 


This phase starts with the approval of the NAPO (NATO production objective). The NAPO is an outline 
statement of the manufacturing process, personnel and facilities required for production of the equipment, 
including an outline production programme based on cost plans, quality control requirements, and the 
stated production specifications. This phase includes the manufacture of a system, sub-system or equipment 
in a plant or factory using series manufacturing techniques. 
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3.7.2 Inputs 


Assuming that the project does not include a combined design and development and production phase the 
inputs can be defined as: 


¢ The project has achieved a level of maturity that will enable the final delivered product to satisfy 
the documented capability as defined in the NATO Staff Target and NATO Staff Requirement. 


3.7.3. Outputs 


Assuming that the project does not include a combined design and development and production phase the 
outputs can be defined as: 


e At the end of production phase the product complies with NATO Staff Requirement Document or 
is supported with an incremental plan achieve the NATO Staff Requirement. 


3.7.4 Life Cycle Costing Benefits 


During the production phase the life cycle cost estimates made on the product components will change 
either because of component obsolescence, change in design or integration with other components/ 
systems. All of these factors will have a direct impact on the Systems Life Cycle Management Plan‘. 
Therefore it is essential that any revised data is collected so that budgets and plans can be amended 
accordingly. 


3.7.5 Types of Life Cycle Cost Studies 


During the production phase the only related life cycle costing study is the review and update of the 
financial element of the systems life cycle management plan. Other life cycle costing studies may be 
initiated if there is radical change brought about by the failure to achieve the forecasted plan or a change in 
policy or procurement strategy by the participating nations in the joint project. 


3.7.6 Methods Employed 


3.7.6.1 Prior to Start of Production 


To ensure that there is a sound basis for tendering/contracting the programme must have completed its 
design and development phase and a full production requirements specification is available. 
The arrangement for work share between participating nations must also be defined. 


To enable suppliers to estimate costs and gain an understanding of the full requirements the acceptance 
standards must also be defined and documented. 


To form a sound basis for both industry and government (including the commitment by the Armed 
Services during tests and trials) to plan their commitment to the project, the participating nations must 
supply a document that defines the equipment and specialist support they intend to supply free of charge 
(government funded equipment and the government funded support known collectively as GFX). 


If there is no competition for the programme (single source supply) government investigating departments 
must be provided with adequate information from the suppliers to enable an assessment to be made of the 
reasonableness of the quotation. Therefore the invitation to tender document should be used to detail all 
the deliverables and acceptance criteria that will enable the authority to make an assessment of the 
contractors bid submission. A typical request for information is shown at Annex B. 


: Policy for Systems Life Cycle Management C-M(2005) 0108-AS1, January 2006. 
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To provide assurance and programme control the bid submission must be soundly based, with key risks to 
performance, cost and timescale identified, and actions planned to monitor, mitigate and control those 
risks. Depending on the circumstances and the number of participating nations consideration should be 
given to the combining of the risk registers prior to tendering. If this cannot be done prior to tendering 
then it is recommended that a joint risk register should be developed as soon as the contract is agreed. 


The contract must meet the national and international requirements for contract law. 


Experience has also shown that in a series production environment a correlation can be established 
between the reduction of production time and the quantities produced. Details on the theory and 
application are given in the example at Section 3.7.9. 


The learning factor here equates to both the production quantity and the spares. Care must be taken to 
ensure that all cost benefits arising from increased quantities (learning) must be taken into account. 


3.7.6.2 During Production 
The primary focus here will be to assess how the forecasted costs compare to the actual cost, particularly 


those that relate to the in-service element. 


The cost element of the systems life cycle management plan should be refined during the course of 
production to reflect refined product and sub component life cycle information (e.g. MTBF, MTTR, etc.). 


Regular joint reviews of the systems life cycle management plan must take place to ensure that impacts 
can be assessed. 


Review the integrated test, evaluation and acceptance plan. This will ensure you that there are adequate 
means to demonstrate that the equipment is fit for service. 


The supplier shall provide progress reports either by EVM (Earned Value Management) or equivalent 
reports. 


Regularly review the risk register and risk mitigation plans. 


3.7.6.3 Post Production 


To inform follow-on projects a post stage evaluation report should be prepared to ensure that best practice 
is captured and any weaknesses for future redress are exposed. 


It is recommended that all actual costs incurred by the contractor are certified. This data can 
subsequently be used to refine and calibrate future cost forecasting models. 


3.7.7 Life Cycle Costing Process 


Figure 3-19 provides a simple illustration of the life cycle costing process. It shows that at this PAPS 
phase the source data to support the life cycle costing is likely to be very mature in terms of system 
engineering design. Quotations for production should be available together with all the logistic 
information required to model the supportability of the system over the expected lifetime. The cost 
breakdown structure should now be fully populated to provide a comprehensive view of the life cycle 
costs. 
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Figure 3-19: Simplified Production Phase Life Cycle Costing Process. 


All the risks (both government and industry) should now be captured and quantified within a joint risk 
register to provide the basis for conducting the cost and schedule risk analysis. There should also be 
mitigation plans for all the risk areas and the costs of undertaking these should also be included in the life 
cycle cost estimates. 


During this phase the life cycle cost output should be a detailed account of all the line items contained in 
the cost breakdown structure. It can be used to support the population of the production activity plan and 
subsequent cost control processes such as earned value management. A comprehensive risk analysis in 
terms of cost and schedule impact should also be available. In summary, the key considerations to the life 
cycle costing process for this phase are: 


The principal data sources will be firm/fixed price quotations from prospective suppliers 
supplemented by military data on likely deployment and staffing numbers. 


All design data and a comprehensive CONOPS report is likely to be available to refine the 
operating and support cost estimates and provide a robust baseline for supportability assessment 
and long term financial budgeting. 


The life cycle costing will be done at a detailed level. 


There will be very few assumptions to be recorded and assessed. Although, it can be expected that 
the reliability and maintainability modelling will have been conducted using predictive data. 
The criticality of this data should be measured using sensitivity analysis. 


The risk will be measured at a detailed level utilising a quantified detailed joint risk register. 


The life cycle costs are likely to be determined at all the line items in the cost breakdown 
structure. 
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3.7.8 Risk Assessment 


During the production phase all the risks will be reviewed and managed on a regular basis. Mitigation 
plans will be put into action and their progress monitored. This will be a comprehensive management and 
analysis activity and will regularly report the possible outcomes in cost and schedule to the project 
manager. 


3.7.9 Example 


An example of cost analysis activities conducted at this phase include the use of cost improvement curves 
also referred to as the learning curve theory. 


According to this theory, as the cumulative production quantity increases, unit production costs decrease. 
This can be due to an increase in workers skill levels, improved production methods, and/or better 
production planning. This effect can be quantified in production cost estimates using a product 
improvement curve, or a learning curve. A 90 % learning curve means that, each time the cumulative 
production quantity doubles, the production time (or, comparatively, the production cost) will be 90 % of 
its value before the doubling occurred. The standard measure of organisational experience in the learning 
curve formulation is the cumulative number of units produced, which is a proxy variable for knowledge 
acquired over production. If unit costs decrease as a function of such knowledge, organisational learning 
in some form is said to occur. 


In actuality, unit (or Crawford’s) learning curves may vary considerably depending on the expected 
magnitude of the cost savings estimated for the 2nd unit and so on. Because of the product complexity, an 
80 % unit cost reduction may not be realised until the production of the following units. Cost analysts 
should evaluate the complexity of tasks in the production process and attempt to determine the type of unit 
learning curve that is most appropriate for the specific situation. 


The equation used in Crawford’s model? is: 


where: 
= time (or cost) required to produce the first unit. 
= the incremental unit time (or cost) of the lot midpoint unit. 


the algebraic midpoint of a specific production batch or lot. 


TKK ® 
ll 


= percent change in cost divided by the percent change in quantity. 
The unit cost of the mid-point unit is the average unit cost for the production lot. 


Sometimes, in the application of learning curves, it may be more appropriate to use a cumulative average 
learning curve. If it turns out that the projected average cost of producing the first 20 units is 80 % of the 
average cost of producing the first 10 units, then the process follows an 80 % cumulative average learning 
curve. 


The theory of the learning curve or experience curve is based on the simple idea that the time required to perform a task decreases as a worker 
gains experience. The basic concept is that the time, or cost, of performing a task decreases at a constant rate as cumulative output doubles. 
Learning curves are useful for preparing cost estimates. 


There are two different learning curve models. The original model was developed by T. P. Wright in 1936 and is referred to as the Cumulative 
Average Model or Wright’s Model. A second model was developed later by a team of researchers at Stanford. Their approach is referred to as 
the Incremental Unit Time (or Cost) Model or Crawford’s Model. 
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In Wright’s model, the learning curve is defined as 
b 
Y=aX 


where: 
= time (or cost) required to produce the first unit. 
= the cumulative average time (or cost) per unit. 


the cumulative number of units produced. 


cM KB 
I 


= percent change in cost divided by the percent change in quantity. 


Figure 3-20 gives an example of a learning curve. 


Wright’s Learning Curve Model 


Y 


Y = aX? 


X 


Figure 3-20: Learning Curve Example. 


3.7.10 References 


[24] www.ams.mod.uk 


[25] Life Cycle Cost Policies Based on Learning Curve Concepts, Lt Col Massimo Pica, Ministry of 
Defence, Directorate General of Land Armaments, Rome, Italy. 


[26] US Air Force Analyst’s Handbook, Christopher A. Feuchter, January 2000. 


[27] The “Mother of all Guesses”, A User-Friendly Guide to Statistical Estimation, Francois Melese and 
David Rose, Armed Forces Comptroller, 1998. 


[28] Unit Cost — a Financial Management Tool for Today and Tomorrow, Comptroller of the Department 
of Defense, 1990. 


[29] U.S. Air Force Life Cycle Management Plan (LCMP) Guide, March 2005 https://acc.dau.mil/ 
GetAttachment.aspx?id=3263 1 &pname=file&aid=6197 
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[30] Statistical Methods for Learning Curves and Cost Analysis, Matthew S. Goldberg and 
Anduin E. Touw, 2003. 


3.8 IN-SERVICE PHASE 


3.8.1 Summary Definition of Phase 


This phase starts with the NISEG (NATO in-service goals). The NISEG is a statement of the general 
utilisation intentions for equipment including reference to national or co-operative logistic and training 
arrangements. 


The NATO PAPS defines the in-service phase as “the operational utilisation of equipment by nations”. 
This utilisation phase is executed to operate the product at the intended operational sites to deliver the 
required services with continued operational and cost effectiveness. 


3.8.2 Inputs 


According to a suitably determined CBS (cost breakdown structure), data related to the operation and 
maintenance activities of a considered system must be collected. An illustrative CBS is shown thus: 


Operation 
Personnel 
Training 
Infrastructure/Facilities 
Consumables 
Others 

Maintenance 
Personnel 
Training 
Facilities 
Spares 
Repair Parts 
Documentation 
Test and support 
Infrastructure 
PHST 
Modification/Upgrading 


Various data formats are used to gather cost and non-cost (e.g. technical) data. Databases which are 
specifically designed for automated information and reporting systems are examples. More information on 
the collection of data can be found in Chapter 6. After data is obtained, data analysis and normalisation 
process must be performed to have a consistent data set (adjustment/normalisation of data). 


3.8.3. Outputs 


The essential output of in-service phase life cycle cost studies is the gathering of actual costs occurring 
during the operation and maintenance activities. However, it should be recognised that there will be 
differences in the expected data due to the approach adopted by the users to maintain the availability of the 
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system during its life. Care is needed therefore to ensure that the gathered data is correctly interpreted and 
properly used in any life cycle cost analysis. 


3.8.4 Life Cycle Cost Benefits 


One of the main objectives of life cycle costing is to provide suitable data in order for the decision makers 
to forecast future costs, manage existing budgets and undertake options analysis where necessary. 
The forecasting of future expenditure requires a sound knowledge of the actual in-service costs. Therefore, 
the collection of actual costs during the in-service phase helps: 


¢ To analyse differences between forecasting and actual costs. 
¢ To help to identify potential areas of cost saving. 

¢ To feed cost databases. 

¢ To identify cost drivers. 

¢ To implement management control. 


¢ To plan to phase out the system and reduce stockholding. 


Life cycle cost studies during this phase are also used to achieve the following objectives: 


¢ One of the main objectives of life cycle cost studies is to provide a forecast of future spending on 
operating and maintenance costs. This will determine the likely financial commitment to support 
the system in-service. 


¢ Monitoring costs: to identify cost growth and inaccuracies. This can be achieved by comparison 
between actual cost and earlier estimates. If unpredicted cost growth is identified, more detailed 
studies that focus on the identified differences has to be undertaken. This is in order to find the 
reasons behind the unexpected cost growth, it may stem from a different actual utilisation than 
was used in earlier cost modelling estimates. This is particularly important if a logistic support 
contract has been awarded with a mechanism of penalties and incentives. 


¢ The life cycle cost studies give operational planning departments the opportunity of estimating the 
costs of activities to be performed and selecting between them, particularly when activity based 
approaches for cost control are followed. Therefore, in-service phase costing enables the discard 
of activities in a cost-effective manner. 


¢ Data collection activities on systems in an operational area supply factual historical data about 
those systems performance. This data can subsequently be used to support future life cycle costing 
studies. 


¢ Involvement in a new investment analysis process anytime there is a significant change to the 
planned in-service programme. 


e The in-service life cycle costs also enables staff to address such questions as: 
¢ Consider a buy or lease option on the services to be provided? 
¢ Determine the level of stock to be held and where are they best placed? 


e What maintenance operations should be undertaken at the various support organisations? 


3.8.5 Types of Life Cycle Cost Studies 


The in-service phase gives the cost analyst the opportunity of verifying the estimates with the actual costs. 
Because of that, the life cycle cost activities during this phase concentrate on the data collection of actual 
data and information. Data collection processes can be accomplished using manual techniques and/or 
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automated information/database systems. Some data analysis must be performed in order to prepare the 
data for use in cost estimating models or cost estimating relationships. 


During the in-service phase, studies are performed in order to refine the life cycle cost estimate of the 
systems in use by using actual recorded data. Further analysis may be done in order to determine the 
failure rates of a system in use and measure the effects of those failures against the predicted values. 
Therefore, it is important to capture system related reliability parameters such as MTBF (mean time 
between failures) and MTTR (mean time to repair) so that the operating and support calculations used in 
the life cycle costs are properly calculated on an engineering rather than a budget controlled basis. 


In addition to the traditional studies conducted during this phase on evaluating cost reduction 
opportunities, studies on the likely costs and implications on the phasing out of the systems are likely to be 
conducted. This will include stock reduction and disposal considerations such as scrap, resale or recycle. 


3.8.6 Methods Employed 


For determination of the in-service phase costs, quantitative expressions of the costs must be specified. 
It falls into three categories. These are: 


¢ Cost of platform/system life-cycle operation. 
¢ Cost of platform/system life-cycle support. 


¢ Cost of platform/system modifications. 


Methods employed during the in-service phase include system dynamics and discrete event simulation to 
provide predictive outcomes, but also the parametric method is used (see Chapter 4 for details). In order to 
capture actual costs methods like activity based costing can be used. As activity based costing provides 
actuals and does not provide any estimates, this methods is not considered in this report. However, 
both type of methods are complimentary and provide the ability to conduct “what-if? scenarios”. 


3.8.7. Life Cycle Costing Process 


Figure 3-21 provides a simple illustration of the life cycle costing process for this phase. It shows that at 
this PAPS phase the source data to support the life cycle costing is likely to be the actual costs arising 
from operating and supporting the system. This information can be used to refine the estimates and 
provide realistic simulation of future spending patterns. The cost breakdown structure should now be fully 
populated to provide a comprehensive view of the life cycle costs. 
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Figure 3-21: Simplified In-Service Phase Life Cycle Costing Process. 


Life cycle cost studies will concentrate on future upkeep or upgrade plans as well as providing future cost 
spending profiles. 


Any risk analysis activity at this phase will concentrate on future support plans such as alternative support 
solutions or planned upkeep/update programmes. As previously, these risks would be captured and 
recorded in a risk register and analysed with respect to the effect on future planning. 


During this phase the life cycle cost output should mirror that of the actual expenditure. The output will 
therefore be used for future budgeting and feedback to support analysis on improving the estimating 
capability and the forecasting of future systems. In summary, the key considerations to the life cycle 
costing process for this phase are: 


¢ The principal data sources are likely to be the actual costs incurred in the operation and support of 
the equipment. This can then be measured against the planned budget and use of the equipment. 


¢ The reliability, availability and maintainability targets can be analysed against the actual effort 
incurred. This will provide a basis for adjusting the supportability plans. 


e The life cycle costing activities are likely to be conducted using detailed cost analysis models 
concentrating on the operation and support functions. 


¢ There will be very few assumptions to be recorded and assessed. Previous assumptions will have 
been superseded by actual data. However, financial assumptions on future escalation, exchange 
rates, etc., will still be made. 


3.8.8 Risk Assessment 


The risk assessment during the in-service phase will more likely support any studies of alternative support 
scenarios. The principles and practice discussed earlier are still valid for this phase. Care will be needed 
however, to counter any over-optimistic predictions when comparing the alternative solutions. It would be 
unwise to introduce any risk into an in-service programme without a comprehensive analysis of the 
possible outcome. 
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3.8.9 Example 


ABC (Activity Based Costing) is a good example of an in-service life cycle cost study for determining 
actual costs. 


ABC is an accounting technique that determines the actual cost associated with products and services 
produced by an organisation without regard to the organisational structure. The method has its roots in the 
manufacturing sector and was developed as a model to identify activities that add value and those which 
are non-added value with the aim of removing waste. 


Activities can be defined as a named process, function or task that occurs over time and has a recognised 
result. Activities use up assigned resources to produce services or products for life cycle costing. 
These can be activities such as repair of equipment, training personnel, and in some cases a combination 
of military activities. 


The system requires a high level of investment in developing recording schemes and element definitions. 


Cost analyses are performed using an in-house model named Educational Cost Effectiveness Analysis 
Model (EDCAM). The interface and menu components of the model are illustrated in Figure 3-22. 
As shown in the figure, the model requires resources, activities, products and the allocation of resources 
and activities to activities and products reciprocally to be defined. The main output of the model is the cost 
of any defined activity based on the resources used, and the product total costs. The model also includes 
an effectiveness estimation module which works on multi-criteria basis. The functional capabilities of the 
model enable the educational planners to select the most cost-effective training programmes. 
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Figure 3-22: The Interface and Menu Components of EDCAM. 


3.8.10 References 
[31] RTO-TR-058 “Cost Structure and Life Cycle Costs for Military Systems” Technical Report. 


[32] Turkish Armed Forces Logistics Concept. 
[33] Turkish Navy ILS Guide. 


[34] www.ams.mod.uk 
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[35] FAA Life Cycle Cost Estimating Handbook, Investment Cost Analysis Branch ASD-410, June 3, 
2002. 


3.9 NATIONAL DISENGAGEMENT 


3.9.1. Summary Definition of Phase 


This phase starts with a NADI (National Disengagement Intention). The NADI is a statement of a decision 
to withdraw equipment from operational utilisation, including forecast dates, quantities and other relevant 
information such as age, condition of equipment, and availability of spare parts. 


However, in some NATO nations, cost estimates for this phase are conducted in the project definition 
phase. The estimates for the likely disengagement cost are required at the early phases in order to meet the 
needs of a government accounting practice whereby a funding provision (included in the life cycle cost 
estimate) accrues annually over the project’s life to equal the value of the predicted financial liability at 
the end of the programme’s life. 


This difference of opinion (do it first — leave it till later) has caused some confusion with respect to what 
should be included in the life cycle costing boundary and when the cost estimates for this phase are best 
conducted. 


For nuclear and other dangerous substances then it is clear that an estimate of the likely disposal costs 
should be included in the early life cycle costing analysis. For other substances where the likely financial 
liabilities are lower then it may be more appropriate to assess these costs at a later stage in the programme. 


The following sub-sections present the view that cost estimates for this phase will be conducted at the 
Project Definition phase and will support an impact assessment using PESTEL (Political, Economic, 
Social, Technological, Environmental and Legal) analysis® or other similar techniques. 


3.9.2 Inputs 


For a preliminary assessment three primary criteria (viability, socio-political considerations and finance 
aspects) are defined, each of which will require evaluation on a range of issues. Examples of the criteria 
normally include: 
¢ Viability to cover the following issues: 

¢ Technical feasibility. 

e« Environmental Impact (technical considerations). 

¢ Legislation/Regulation. 

e Flexibility to change. 

e Time sensitivity. 


¢ Socio-political considerations: 
¢ Response from Pressure Groups. 
¢ Public Acceptability/Concerns. 
e Acceptability to Parliamentary Groups. 
e« Environmental Impact (site). 


6 Le . : : : 
PESTEL: Political factors, Economic factors, Social factors, Technological factors, Environmental factors, Legal factors. For each set of 
factors, you need to evaluate what is the situation and what developments are likely to take place in the next future. 
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e Alignment with Government policy. 
e Alignment with International policy. 
e Historical acceptability. 


¢ Physical security constraints. 


¢ Financial aspects including: 
¢ Overall cost. 
¢ Commitment and spend profile. 
¢ Risk of extra cost materialising. 
e Affordability. 


3.9.3. Outputs 


Each of these would have an associated critical success factor and the disposal options can then be 
measured and evaluated. 


3.9.4 Life Cycle Cost Benefits 


The long term plan should include the identification and consideration of a number of disposal options. 
By conducting life cycle cost analysis the following options can be assessed: 


¢ Re-deployment (can the equipment be used for training/instructional use, as a heritage/museum 
asset, for spare recovery, etc.). 


¢ Reclamation, recycling, re-manufacture (is there a possible other use as opposed to disposal). 
e Sale (potential customers). 


e Disposal at cost. 


3.9.5 Types of Life Cycle Cost Studies 


Some nations are conducting cost studies and developing estimates of the likely disposal costs during the 
concept and feasibility phases in order to inform and make budget provisions. The types of costing studies 
being conducted at this early phase are predominantly restricted to high-level estimates to support disposal 
option analysis and financial liability studies. In this instance, the life cycle cost would be one of the 
criteria measures in a rigorous assessment of the alternative options. For larger programmes where a 
PESTEL analysis is conducted at a later stage then the life cycle costing study would support the financial 
aspects taking cognisance of the results from the other measurement criteria. 


3.9.6 Methods Employed 


Predicting cost estimates for equipment disposal is not dissimilar to the methods employed at the 
pre-feasibility phase. The most common methods of estimating the likely disposal cost currently used is by 
analogy and parametric. These methods are well established and can calculate the negative and positive 
financial impact depending on the alternative disposal options being assessed. However, care should be 
exercised when employing both methods. To avoid error, it is essential that as much historical data as 
possible is gathered and evaluated to provide a degree of ‘normalisation’ such that a ‘like for like’ basis is 
achieved. 
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3.9.7 Life Cycle Costing Process 


Figure 3-23 provides a simple illustration of the life cycle costing process. It shows that at this PAPS 
phase the source data to support the life cycle costing is likely to be immature therefore a greater reliance 
on the types of data sources indicated should be expected. In addition, there will be a high level of 
assumptions in terms of the likely performance/design parameters of the systems being evaluated. Risk is 
likely to be qualitative rather than quantitative. 
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Figure 3-23: Simplified Disengagement Phase Life Cycle Costing Process. 


It should be expected therefore that the life cycle cost output for the disengagement phase will be an 
understanding of the programme liability costs and the level of uncertainty surrounding them. In summary, 
the key considerations to the life cycle costing process for this phase are: 


¢ The principal data sources are likely to be parametric based and for speed and ease of use a 
proprietary cost estimating model is likely to be employed. 


e There will be many assumptions to be recorded and assessed. 


e The risk will be measured at the very top level probably utilising some high level risk register and 
PESTEL techniques. 


¢ The disposal phase costs are likely to be in the form of rough order of magnitude determined by 
legislation, regulation and market forces at the time. 


3.9.8 Risk Assessment 


The identification of risk for this phase will be conducted at a very high level. It is likely to be a 
combination of single line statements and will probably contain a mixture of issues as well as risks. 
The cost analyst will need to distinguish the difference between them in order to ensure that only those 
applicable risks are to be included in the cost estimate. 


Where there is no risk register or risk record then another approach would be to employ an optimism bias 


technique. Here it could be employed to redress overly optimistic tendencies by making empirically based 
adjustments to the cost estimates (this technique is explained at Chapter 7). 


3 - 46 RTO-TR-SAS-054 


LIFE CYCLE COSTING IN THE PAPS PHASES 


At the very minimum, the life cycle cost estimate produced at this phase should include or indicate the 
level of financial risk exposure and liabilities. 


3.9.9 Example 


There are significant benefits and cost saving opportunities to be realised by reducing asset holdings. 
Current approaches include the use of marketing arrangements to examine the most cost effective disposal 
method. 


The principal means of gaining a return on surplus inventory is either by scrap or sale. By definition, any 
surplus equipment or stores (excluding nuclear) will have been written off by the user organisation and 
therefore, in theory, are only worth their scrap value. In practice, they are a source of affordable and 
proven defence equipment for other countries and can provide the organisation with an additional source 
of income to offset any future spending. 


As the costs of disposal is therefore determined by the market forces and opportunities at the time then the 
financial aspects of the PESTEL criteria should reflect the implication and cost associated with scrap value 
only. 


During the in-service phase further opportunities such as re-deployment or sale may arise. These should be 
seen as an opportunity. A full cost benefit analysis (see Sub-section 3.3.9.2) should then be undertaken to 
support the decision process. 


3.9.10 References 
[36] Middleton, J., “The Ultimate Strategy Library”, Capstone Reference. 


[37] UK MoD, Operational Analysis — Foundation for the Business Case, November 2003, Director 
General (Scrutiny and Analysis) [unpublished MoD Documents/Reports]. 


[38] www.ams.mod.uk 


3.10 NATO POLICY FOR SYSTEMS LIFE CYCLE MANAGEMENT AND THE 
FUTURE OF PAPS 


3.10.1 General Comment 


As stated previously, the PAPS life cycle phases have been adopted in this report because they represent a 
commonly accepted NATO standard. Furthermore, they are reasonably well defined and they form a 
natural framework for extending the process descriptions and guidelines given in PAPS to include costing 
activities. 


It is recognised by the authors of this report, however, that while the subject and purpose of PAPS remains 
as relevant as ever, the document itself, though still in effect, is dated. Many references to NATO 
documents, agencies, etc., are no longer relevant. More importantly, NATO policy has evolved. NATO 
policy for standardisation now calls for the use of civil standards to the maximum practicable extent. Such 
a standard was published in 2002 as ISO 15288 on system life cycle processes. This standard has been 
adopted in NATO policy for SLCM (systems life cycle management). 


At the time of writing this report, a working group under AC/327 NATO LCMG (life cycle management 
group) is working to update PAPS to comply with NATO policy for SLCM and the related AAP-48 
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on NATO life cycle stages and processes, which is a guide to implementing NATO policy for SLCM. 
(AAP-48 is, at the time of the writing of this report, in the draft stage.) 


While it is therefore likely that the PAPS will be adapted to comply with ISO 15288, the original PAPS 
life cycle phase definitions are retained in this report. The rationale behind this is that the present 
definition is still NATO practice and that the focus of the original PAPS on the early part of the life cycle 
is more in line with objectives of this report. 


ISO 15288 divides the system life cycle into six stages, defined later in this chapter, as opposed to the 
eight phases of the original PAPS. The main difference between them is that PAPS treats the earliest part 
of the life cycle in more detail, while ISO 15288 divides the in-service activities into utilisation and 
support activities. Note that mission need evaluation, the first phase of the PAPS life cycle model, is not 
considered part of the system life cycle in ISO 15288. A rough translation between the original PAPS life 
cycle phases and the ISO 15288 life cycle stages is provided in Figure 3-24. 


Mission Need Evaluation 


Prefeasibility 


Feasibility Concept 


Project Definition 


Design and Development ey Development 


pees | 
Production 


Utilisation 
In-Service —— 
Support 
Disengagement gy Retirement 


Figure 3-24: “Translation” from PAPS Phases to ISO 15288 Stages. 


3.10.2 ISO 15288 Life Cycle Stage Definitions 


AAP-48, NATO life cycle stages and processes is based on ISO 15288. The stages defined in AAP-48 are 
the same as those included in ISO 15288 AAP-48. The following definitions are therefore taken from 
AAP-48. 


3.10.2.1 Concept Stage 
The concept stage starts after the decision to fill a capability gap with a materiel solution and ends with the 


requirements specification for this materiel solution. The purpose is to evaluate the relative need, potential 
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risks, and cost benefit of a proposed system, or a major upgrade of an existing system prior to any 
commitment of resources. One or more alternative solutions to meet the identified need or concept are 
developed through analysis, feasibility evaluations, estimation (such as cost, schedule, market intelligence 
and logistics), trade-off studies, and experimental or prototype development and demonstration. 


3.10.2.2 Development Stage 


The development stage is executed to develop a SOI (system of interest) that meets user requirements and 
can be produced, tested, evaluated, operated, supported and retired. This stage also ensures that the aspects 
of future stages (production, utilisation, support, and retirement) are considered and incorporated into the 
design through the involvement of all stakeholders. 


3.10.2.3 Production Stage 


The production stage is executed to produce or manufacture the product, to test the product and to produce 
related supporting and enabling systems as needed. 


3.10.2.4 Utilisation Stage 


The utilisation stage is executed to operate the product at the intended operational sites to deliver the 
required services with continued operational and cost effectiveness. This stage ends when the SOI is taken 
out of service. 


3.10.2.5 Support Stage 


The support stage is executed to provide logistics, maintenance, and support services that enable continued 
SOI operation and a sustainable service. The support stage is completed with the retirement of the SOI and 
termination of support services. 


3.10.2.6 Retirement Stage 


The retirement stage provides for the removal of a SOI and related operational and support services and to 
operate and support the retirement system itself. This stage begins when a SOI is taken out of service. 


3.10.3 References 


[39] NATO International Staff — Defence Support Division, Allied Administrative Publication (AAP) — 
20 — Handbook on the Phased Armaments Programming System (PAPS), February 1989. 


[40] C-M(2000)54 — NATO Policy for Standardization, 20 September 2000. 


[41] ISOAEC (nternational Standards Organisation/International Electrotechnical Commission) 15288 
“Systems Engineering — System Life Cycle Processes”, 1 November 2002. 


[42] C-M(2005) 0108-AS1 — NATO Policy for Systems Life Cycle Management, January 2006. 


[43] PFP(AC/327)D2006(0009) — Allied Administrative Publication (AAP) — 48 — on NATO System Life 
Cycle Stages and Processes (Draft Edition 1), February 2006. 
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Chapter 4- METHODS 


4.1 GENERAL 


This chapter discusses the different methods of generating life cycle cost estimates and conducting cost 
analyses at various stages in the NATO PAPS cycle. It is not meant to be a prescriptive description of the 
methods which are best found by accessing the reference books and web sites but does provide guidance 
on the appropriate approaches to life cycle costing for each PAPS phase. 


4.2 OVERVIEW OF METHODS 


Most cost estimates require the use of a variety of methods. A different approach may be used for each 
area of the estimate so that the total system methodology represents a combination of methods. Sometimes 
a second method may be used to validate the estimate. 


When choosing an estimating method, the cost estimator must always remember that cost estimating is a 
forecast of future costs based on a logical interpretation of available data. Therefore, availability of data 
will be a major factor in the estimator’s choice of estimating methodology. 


The best combination of estimating methods is the one which makes the best possible use of the most 
recent and applicable historical data and systems description information and which follows sound logic to 
extrapolate from historical cost data to estimated costs for future activities. 


An example of this is would be to use data gathered through expert opinion combined with methods 
for simulation to obtain reliable data to conduct simulations on different support organisations. 
Linear programming might then be used to optimise a spares inventory for the chosen support 
organisation. These values can then be used in the parametric techniques employed in estimating the total 
life cycle costs for the programme. 


The following table shows how the methods have been categorised for easy reference. 


Table 4-1: Method Categorisation 


Method Category Methods 

Optimisation Linear programming 
Heuristics 

Simulation System Dynamics 


Discrete Event 

Monte Carlo 
Calculation/Estimation Analogy 

Parametric 

Bayesian 

Engineering 

Catalogue 

Rule of Thumb 

Expert Opinion 

Decision Support Analytical Hierarchy Process 
Multi-Criteria Decision Analysis 
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4.2.1 Optimisation Methods 


Mathematical programming and heuristics are both common forms of optimisation methods. Linear 
programming is a subset of mathematical programming but deemed important enough to be described 
separately. 


4.2.1.1 Linear Programming 


Linear programming is a mathematical modelling technique designed to optimise the usage of limited 
resources. The usefulness of this technique is enhanced by the availability of highly efficient computer 
codes. A linear programming model consists of three basic elements: 


¢ Decision variables that need to be determined. 
¢ Objective (goal) that need to be optimised. 


¢ Constraints that need to be satisfied. 


Linear programming is particularly useful for large and medium scale problems in which there are many 
variables and many constraints to be considered. Therefore, the use of linear programming is often supported 
by computer software. 


4.2.1.2 Heuristics 


Methods based on heuristic approaches use standardised rules of thumb repeated many times in order to 
find a good enough solution to a problem. These types of models can be easier to apply than the 
mathematical programming methods. There are, one the other hand, no guarantees that the solutions found 
will be the optimal choices for solving the problem. 


4.2.2 Simulation Methods 


System dynamics and discrete event simulation are both forms of simulation models that allow a 
representation of the activities of a system over time. In each case, the simulations step through time and 
perform calculations for that point in time which will change the state of the system in some way. The end 
state at one point in time is the start state for the next. 


4.2.2.1 System Dynamics 


System dynamics works by using even time steps. It keeps track of how many items are in particular 
locations (stocks) in the system (items can be entities such as people, cash or can represent fluids). 
It works by allowing flow into and out of stocks through valves. The structure of the model allows the 
development of behaviour to control the flows, and to provide measures based on the state of the system 
(such as costs). One of the most powerful features of system dynamics is the visual structure of the model 
that helps users and developers to understand the relationships between elements of the model. 
This structure allows the representation of complex behaviour while using comparatively simple equations 
for each relationship. 


System dynamics are usually good for building models with a wide scope and long run behaviours. 
They are generally quicker to build than discrete event simulations and also execute more quickly. 
The models do not usually contain stochastic elements although they can be repeatedly run with different 
input values to examine uncertainty around inputs. System dynamics models are good for life cycle 
costing where there can be a wide scope for cost drivers, large numbers of items and long duration. 
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4.2.2.2 Discrete Event Simulation 


Discrete event simulation uses uneven steps in time with the model jumping to the point in time where the 
next event will occur. The event will cause a change in state of the system that can trigger other events to 
occur immediately and/or schedule another event to occur at a point in the future. The model keeps track 
of every entity in the system in terms of location and can store characteristics of each entity. Many of the 
models have animations that show the state of the system, although much of the actual logic is hidden 
below the surface of the model. 


Discrete event simulation is good for building models with a narrow scope and relatively short-term 
durations. They generally take longer to build than system dynamics models and execute more slowly 
because each of the entities is represented individually. The model allows stochastic elements, using 
sampling from probability distributions to represent things like inter-arrival times and durations of 
activities. Due to the stochastic elements in the models, all experiments should make use of multiple runs 
in order to calculate means and standard deviations for the key output variables. Discrete event simulation 
is good for logistics models where it is important to understand how the system can deal with peaks and 
troughs in demand. 


4.2.2.3 Monte Carlo Simulation 


Monte Carlo simulation is used in defence cost analysis to generate frequency or probability distributions 
which are otherwise too difficult or impossible to generate mathematically, that is, using formulae. 
More specifically, all variables in a cost estimating model potentially affected by risk and uncertainty are 
first identified. Then, probability distributions are estimated or selected for each. This entails first 
choosing the type of distribution to apply and then estimating the distribution’s parameters. Possible 
distribution types include: 


A 4 HE A 


Normal Triangular Uniform Lognormal 


Figure 4-1: Example of Typical Distribution Types. 


Monte Carlo simulation generates random values for each of the uncertain variables over and over again, 
according to the type of distribution chosen, to produce a frequency or probability distribution of total 
costs for a weapon system or automated information system acquisition programme. Figure 4-2 shows a 
typical Monte Carlo output, based on 5000 selections or trials. 
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Crystal Ball Report 
Simulation started on 1/3/01 at 18:24:18 
Simulation stopped on 1/3/01 at 18:24:47 


Forecast: 


Statistics: 


Trials 5000 
Mean 172.12 
Median 172.33 
Mode oo 
Standard Deviation 43.32 
Variance 1877.03 
Skewness 0.01 
Kurtosis 2.88 
Coeff. of Variability 0.25 
Range Minimum 22.38 
Range N\ : 94 
Range \ Forecast: 56 
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Figure 4-2: Example of Monte Carlo Output. 


4.2.3. Calculation/Estimation Methods 


4.2.3.1 Analogy 


The analogous or comparative method assumes that no new programme represents a totally new system. 
Most new programmes originate or are evolved from already existing or simply represent a new 
combination of existing components. 


The analogous method compares a new system with one or more existing systems for which there are 
accurate cost and technical data. The historic system should be of similar size, complexity and scope. 
The estimator/analyst makes a subjective evaluation of the differences between the new system of interest 
and historic systems. Normally, engineers are asked to make the technical evaluation of the differences 
between the systems. Based on the engineer’s evaluation, the cost estimator/analyst must assess the cost 
impact of the technical differences. 


It is not necessary to compare the new system to just one other analogous system. It may be desirable to 
compare some sub-systems of the new system to sub-systems of old system A, and other to sub-systems of 
old system B. 


The advantage of the analogy method is that if a good analogy can be found, it allows for a lower level of 
detail, thus enhancing the credibility of the estimate. 


The estimator should be cautious of using this technique without fully understanding the basis and the 


proper usage context. The major disadvantage of the analogy method is that it can be difficult to find a 
good analogy and the required engineering judgment. 
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An example of this method can be found at Sub-section 3.5.9.1. 


4.2.3.2 Parametric 


The parametric method estimates costs based upon various characteristics or measurable attributes of the 
system, hardware and software being estimated. It depends upon the existence of a causal relationship 
between system costs and these parameters. Such relationships, known as CERs (cost estimating 
relationships), are typically estimated from historical data using statistical techniques. If such a 
relationship can be established, the CER will capture the relationship in mathematical terms relating cost 
as the dependent variable to one or more independent variables. Examples would be estimating costs as a 
function of such parameters as equipment weight, vehicle payload or maximum speed, number of units to 
be produced or number of software lines of code to be written. The CER describes how a product’s 
physical, performance and programmatic characteristics affect its cost and schedule. 


The parametric or statistical method uses regression analysis of a database of similar systems to develop 
the CERs. Therefore parametrics rely on complex relationships and therefore require a considerable 
amount of data to accurately calibrate. Some of the commercially available cost estimating models do 
have historic public domain information attached and this enables the model to achieve reasonable results 
in the early phases of the procurement cycle when capability is known, but detailed requirements are 
poorly defined. 


Parametric estimating is used widely in government and industry because it can easily be used to evaluate 
the cost effects of changes in design, performance, and programme characteristics. The major advantage of 
the parametric method is that it can capture major portions of an estimate quickly and with limited 
information. Parametric cost estimating, in essence, is usually a form of hedonistic regression analysis. 
More specifically, the cost of a weapon system, or component thereof, is typically postulated as a function 
of the technical, performance, and programmatic characteristics of that system. There are several 
advantages of this cost estimating technique: 


* Objectivity. The cost-estimating relationship, ideally, is based on consistent, quantitative, 
non-subjective inputs, or values of the dependent and explanatory variables. 


¢ Ease of Use. Values of cost, the dependent variable, can be easily calculated based on changes to 
any of the explanatory variables. This is useful for what-if, sensitivity analyses. 


¢ Tests of validity. Standard outputs of regression analysis include F and t statistics which measure, 
respectively, the overall power of the set of explanatory variables in explaining changes in costs 
and of the significance of any one variable in explaining changes in costs. 


A critical consideration in parametric cost estimating is the similarity of the systems in the underlying 
database, both to each other and to the system which is being estimated. Additionally, the database must 
be homogenous. A data element entry for one system must be consistent with the same data element entry 
for every other system included in the database. The major disadvantage of the parametric method is that it 
may not provide low level visibility and subtle changes in sub-elements cannot be reflected in the estimate 
easily. 


An example of this method can be found at Sub-section 3.5.9.2. 


4.2.3.3 Bayesian Techniques 


Bayesian techniques deal with how a prior belief should be modified in the light of additional information 
e.g. later information or information from another source. A parameter to be estimated is known, on the 
basis of information available at the time, to have a certain value subject, since that information is 
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incomplete or of a probabilistic character, to a range of uncertainty i.e. there is a “prior probability 
distribution” of that parameter. Further information then becomes available which, of itself, suggests a 
different value and probability distribution for the parameter. Bayesian inference allows these two sets of 
data to be combined to give the most probable value (and least uncertainty) for the parameter in question 
i.e. the correct “posterior distribution’. 


Beliefs are expressed either as the probabilities of a finite number of discrete outcomes of a future event or 
else, as here, as the probability distribution of a continuous variable. The question to be answered 
becomes, therefore, that of how an initial estimate (the ‘prior’ distribution) is best modified in the light of 
additional information so as to obtain a refined estimate (the ‘posterior’ distribution). 


Figure 4-3 presents the relationship between the inputs and outputs and shows how the cost estimates can 
be based on that which is known with some certainty and not on what can only be conjectured at the time 
the estimates are made. The approach also provides performance-based estimates from the earliest stage of 
the project life cycle and allows more precise design-based estimates to be derived as proven design data 
becomes available. 


Inputs 


Performance 
requirements 


Design norms for 
ieeXo LUI e=Yom of=Vacolauit-larer= 
ATi damexe) avite(=varer-m nati ey) 


Design details 
(ECE WET EL e)(<) 
and with 
Corey ayite(=)avex=) 
limits) 


Estimated cost 
of the project 


Outputs 


Figure 4-3: Bayesian Application to Cost Estimating. 


Within the context of the techniques described above, such questions arise from design variables 
(on which the cost estimates are later to be based) being both supplied by the estimator and also derived 
within the model from the performance required of the equipment whose costs are to be estimated, as also 
input by the estimator. How this is done may be illustrated through an example. 


Suppose that the estimator has supplied an estimate of the displacement of a ship as to be 10,000 + 3,000 
tons. This is the initial estimate i.e. the ‘prior’ distribution of displacement. The estimator has also 
supplied information concerning the performance required of the vessel in question. From that, the model 
computes, as a design norm, a displacement of 12,000 + 4,000 tons. 


It is now necessary to examine the compatibility of these two estimates. 


4-6 RTO-TR-SAS-054 


METHODS 


In this case, the two estimates of displacement do not conflict. Rather, there is a high probability that they 
are both estimates of the same quantity i.e. of what will be the displacement of the vessel when designed 
fully and built. The question at issue becomes then what value of displacement should be used for the 
purposes of estimating the cost of this ship i.e. what ‘posterior distribution should be taken forward into 
the next stages of the calculation. 


There are various possibilities. They are: 
(a) To use the estimate of displacement provided by the model i.e. that of 12,000 + 4,000 tons; 
(b) To use the estimate input by the estimator i.e. that of 10,000 + 3,000 tons; 
(c) To average the two estimates; or 


(d) To combine the two estimates but to weight each according to its reliability as manifest by the 
uncertainty attached to each. 


Clearly, the first approach (a) is incorrect since it concentrates solely on the estimate of lesser certainty 
and ignores that which is more certain. 


The second approach (b) is more reasonable but unsatisfactory. The higher estimate is only somewhat less 
certain and it is inappropriate, therefore, to ignore entirely the indication given by the model that the 
displacement may well turn out to be higher than is supposed at present. 


The third possibility (c) is yet more reasonable but it is still to be criticised. To average the two estimates 
(obtaining, thereby, a figure of 11,000 + 2,500 tons) is to attach equal weight to both even though one is 
less certain than the other. 


The fourth (d) (and Bayesian) approach is optimal. Weights attached to each estimate are those, which 
minimise uncertainty of the combined estimate and, so, make best use of all of the information available. 


Details of the mathematics involved are not repeated here. However, the reader may gain an understanding 
of their basis by regarding each estimate as being (independently and hypothetically) the result of repeated 
sampling from the same (infinite) population comprising all possible values for, in this case, the 
displacement of the vessel in question. The estimate having the greater certainty is then the result of 
averaging more samples than was the case for the result having the lesser certainty. Accordingly the 
former is accorded more weight when all of the samples are pooled and a grand average computed. 


In the present example the result of this Bayesian approach is an estimate of displacement of 10,720 + 
2,400 tons. Note that, as expected and as is reasonable, this inclines somewhat more to the more precise of 
the estimates being combined than their simple average. Note also that the uncertainty of this estimate is 
somewhat less than that resulting from simple averaging, again reflecting optimal use of all of the 
information available. 


The utility of this approach may be illustrated further by considering the evolution of a project. At its 
earliest stages, prior to any design or development work, estimates of design characteristics, supplied by 
the estimator cannot be anything but imprecise. Through the Bayesian approach, the model will then rely 
upon the (the more certain) design norms, which it generates. As design and development proceed more 
certain information will become available to the estimator for input to the model and estimates will be 
based progressively more upon such data. When design and development are complete design 
characteristics will be known exactly. The model will then rely upon those alone thus a single model is 
able to respond appropriately, optimally and automatically to all of the circumstances encountered 
throughout the evolution of a project. 
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4.2.3.4 Engineering (bottom up) 


The engineering or bottom up method of cost analysis is the most detailed of all the techniques and the 
most costly to implement. This technique starts at the lowest level of definable work within in the cost 
breakdown structure and builds up to a total cost. This type of estimate is used when detailed design data 
is available on the system. Two types of engineering estimates can be distinguished: 


e An engineering estimate provided by a contractor. Making sure that the contractor has provided 
all the data and supporting information to clearly define the basis of the estimate. 


e An engineering estimate provided by government personnel (an in-house prepared engineering 
estimate). 


Engineering estimates prepared by contractors differ substantially from engineering estimates performed 
by government in at least two ways. First, the contractor prepared estimate is based on input from work 
units that will do the work and that have performed similar work in the past. Second, contractors are able 
to bring more detailed programme description data to the cost estimating process. For an engineering 
estimate provided by a contractor, an industrial engineer will estimate the labour hours, raw materials and 
parts required to complete the work. The industrial engineer may use a variety of techniques in estimating 
the direct labour and material cost of each discrete work element. 


An engineering estimate prepared by a contractor do not usually include such elements as other 
government costs (e.g. other system and sub-system integration costs). It is also important to ensure that 
any engineering change costs are included in the government budget estimate submissions. 


A contractor prepared engineering estimate will be used or evaluated by a government cost estimator. 
The following guidelines have proven useful in the past with respect to evaluating contractor prepared 
engineering estimates: 


¢ Quickly find out the high cost areas or items and focus attention on them. 


e If the evaluation is part of a source selection compare costs among contractors to spot unusually 
high or low costs for further investigation. 


e If in time more than one cost estimate has been provided by the contractor see whether major 
changes were made to the cost estimate. 


e Use audit report to check the validity of the rate and factors used by the contractor. 


¢ In high cost areas, make sure the contractor has provided all substantiating information requested 
to generate a cost estimate. 


Perhaps the most important guidance here is to require the contractor to submit cost data and 
substantiating information in a format that is clear, complete and ready for evaluation. The NATO 
generic cost breakdown structure developed by SAS-028 may help here. 


In-house engineering estimates are mainly prepared to forecast out year cost for new systems. Government 
cost estimators usually obtain the necessary data through visits to and discussion with the prime 
contractors. 


In-house engineering cost estimates differ from contractor prepared engineering cost estimates in several 
ways. For an in-house estimate fewer estimators, specialists and less information is available, especially 
when prior to production, when not much actual data is available. When the programme is in production, 
the differences should not be so significant. 
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The engineering cost estimate is most often used during the production an deployment phase. 
This technique encourages the contractor to do his homework early on and define all the work down to the 
lowest level of the cost breakdown structure. 


4.2.3.5 Catalogue or Handbook Estimates 


Handbooks, catalogues and other reference books are published that contain lists of off-the-shelf or 
standard items with price lists or labour estimates. The estimator can use these catalogue prices directly as 
unit values for standard components within a larger system. 


4.2.3.6 Rules of Thumb 


These refer to simple usually deterministic cost relationships. They are developed from an analysis of 
existing cost information. 


Any rules developed should only be used at the early stages of project when actual specification and 
requirements are poorly defined. 


4.2.3.7 Expert Opinion 


An expert opinion may be used, when data required to use other techniques is not available. It is a 
judgemental estimate performed by an expert in the area to be estimated. Several specialists can be 
consulted until a consensus cost estimate is established. Surveying a number of experts independently to 
reach a consensus of opinion, the Delphi technique also may be used to provide a collective opinion. 


An expert opinion can also be used to validate an estimate. 


4.2.4 Decision Support Methods 


4.2.4.1 Analytical Hierarchy Process 


In grading or ordering the importance of a number of items in defence decision making, such as lists of 
operational tasks or lists of strategic requirements, Kenneth Arrow’s “Impossibility Theorem” comes into 
play.' In a nutshell, the theorem indicates that no analytical technique exists that will simultaneously 
satisfy all commonly regarded fairness criteria in rank-ordering items in a list. Literally dozens of 
techniques for ordering preferences have been developed over the ages. These include the method of 
pairwise comparisons (used universally in the defence and commercial sectors), Borda’s procedure 
(used by major league baseball in the U.S. for yearly selection of its most valuable player), and Tukey’s 
algorithm, to name just a few. All techniques, however, as Arrow demonstrated, fall short of perfection. 


Nevertheless, the demand for making selections and for ordering preferences remains unlimited. Hence, 
it is important to choose a method of ordering with good, robust statistical properties, such as those 
indicated above or below, realising, of course, that no technique is perfect. 


The Analytic Hierarchy Process (see also ‘Saaty Method’’) process is known as a soft operational research 
approach to quantify how important a criterion is compared to other criterion. This enables acquisition 
decisions to be approached using an auditable method that considers the importance of all the options 
against specific subjective and objective acquisition requirements. 


' Arrow co-shared the Nobel Prize in Economics in 1972 for this work, which was first undertaken in his Ph.D. dissertation a 
couple of decades earlier. 


° The concept of AHP was developed, amongst other theories, by Thomas Saaty, an American mathematician working at the 
University of Pittsburgh. 
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It is used when making complex decisions involving many criteria. The process is particularly useful when 
conducting portfolio and options analysis. Some of the more complex models can provide a three 
dimensional view of the performance, cost and time aspects and present a graphical as well as a tabular 
output. 


As the technique requires subjective judgement it is recommended that the process of allocating 
weightings and scorings should involve a team to avoid bias selections from any individual. 


4.2.4.2. Multi-Criteria Decision Analysis 


MCDA (Multi-Criteria Decision Analysis) is an established operational research technique with wide 
applicability. For example, in the UK, MACE (Multi-Attribute Choice Elucidation) is an adaptation of 
MCDA. It is a method for applying objective measurement to the relative merits of mutually exclusive 
acquisition options. Its principal application is in the assessment of bidder responses to tenders. 
The application of MACE should be focussed on the offer being made by a bidder. In certain 
circumstances investment appraisal may also play a part in the tender option assessment. 


MACE translates key issues from the requirements for the options to be considered into logical items 
known as criteria. For each criterion MACE derives a numerical worth. The intermediate result is an 
assessment hierarchy of clearly defined and measurable criteria which is included in the RFI/ITT. 
Typically, each key user requirement in a NATO staff requirement is a candidate criterion. 


Options are objectively marked against the criterion. Individual criterion marks are transformed and 
aggregated to produce numerical overall merit(s) for each option. The overall, and intermediate, merits are 
compared across the options, so informing the selection process. 


MACE provides a methodical, objective, value adding, defensible and auditable assessment method, but it 
is only an aid to the decision making process. MACE may not always unambiguously isolate the best 
option, but when it does not it will provide reliable information to inform and support option selection. 
The ultimate decision on which option is to be selected is dependent on many factors, possibly including 
assessments using other methods. The factors (e.g. technical, commercial, financial, programme/risk 
management) to be included within a MACE assessment are determined on a case-by-case basis. 
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SUMMARY OF FINDINGS 


This section describes a summary of the methods that are being used by the participating nations, based on 
the analysis of the matrices, introduced in Chapter 1, that were completed by the participants. Figure 4.4 
shows the results of this analysis graphically. 
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Figure 4-4: Summary of Methods Used by the Nations. 


The findings at Figure 4-4 clearly show that to generate a cost estimate all participating nations use many 
methods across each of the phases considered. 


Looking at the categories of methods distinguished in this chapter, the calculation/estimation category is 
used in all phases. The analogy and parametric method are predominant and are used in (almost) every 
particular phase. 


The engineering or bottom-up method is most popular in the phases (project definition, design and 
development and production) when major alternatives are compared and more detailed information is 
available. 


In the very early phases decision support methods and system dynamics are becoming more popular. 
This is not surprising as these techniques can be employed using subjective judgement thus overcoming 
the lack of quantitative historical data. 


In the design and development phase, the production phase and the in-service phase, simulation and 
optimisation methods are sometimes used to estimate support costs and the effects of alternative support 
scenarios. Not shown in the figure, but during the in-service phase activity based costing is widely used to 
capture actual costs. 


4.4 RECOMMENDATIONS 


Many cost estimates require the use of a variety of methods. It is often not possible to use a single method 
to estimate all the cost elements to be considered. Therefore the total life cycle cost estimate of a system 
will include the use and outputs from a combination of methods. 


As shown in Figure 4-4 the participating nations use many different methods in each phase. It is therefore 
not possible to recommend a single method to estimate the life cycle costs for each phase of the life cycle. 


The best cost estimating method is one that makes the best use of the data available. It is therefore 


recommended to employ a method that will provide as much detail as the availability of the input data will 
allow. Therefore, the availability of data is a major factor in the estimator’s choice of estimating method. 
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It is also recommend that a second method is used in order to improve the confidence and to validate the 
life cycle cost estimate. In many cases, expert opinion or a simple rule of thumb can provide a good 
second estimate. 


For multi-national programmes it is important that the method chosen can be used by all the nations 
involved, given the data available in each nation. This will probably result in a method being chosen that 


does not demand detailed design information and supporting data. 


It is recommended that research be conducted continuously to enhance methods and models for life 
cycle costing. 


Periodically, the US Department of Defence undertake an initiative to review the basis and techniques 
employed in cost estimating. This is supported by a number of academic groups and learned societies. 


However, these initiatives purely examine techniques that will be employed within the US. It would be 
beneficial to conduct a similar continual review across NATO and PfP nations. 


4.4.1 References 
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Trade-Offs, Wiley, New York. 


[71] Olson, D. (1995), Decision Aids for Selection Problems, Springer Verlag, New York. 


[72] Yoon, K.P. and Hwang, C.-L. (1995), Multi-Attribute Decision Making, Sage, Beverley Hills. 
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Chapter 5 - MODELS 


5.1 GENERAL 


This chapter presents the results of the matrix survey, mentioned in Chapter 1, on models. The focus here 
is to summarise the use of models by the nations (both commercially available and in-house) and present 
the results in a general overview. The models used and any requirements on the use of a specific model 
across all of the PAPS phases will be presented. Annex A contains all the participating nations’ completed 
matrices. 


In everyday cost analysis language, the terms models and tools are often used interchangeably. This is, for 
instance, exemplified in the questions put forward in the matrix that the nations in the Task Group where 
requested to answer. It might be of interest for some other types of studies to make a distinction between 
the two, but for the scope of this work it has been decided that no such distinction is needed. In the 
Programme of Work for the SAS-054 Task Group the term (cost) model is defined as: 


“A Cost Model: is a set of mathematical and/or statistical relationships arranged in a systematic 
sequence to formulate a cost methodology in which outputs, namely cost estimates, are derived 
from inputs. These inputs comprise a_ series of equations, ground rules, assumptions, 
relationships, constants, and variables, which describe and define the situation or condition being 
studied. Cost models can vary from a simple one- formula model to an extremely complex model 
that involves hundreds or even thousands of calculations. A cost model is therefore an abstraction 
of reality, which can be the whole or part of a life cycle cost.” 


Using this definition, both a graphic description of the relationships that represent the abstraction or 
simplification of reality as well as a series of connected, specially developed computer programmes, 
can be a model.' The bulk of the models discussed in this chapter are of the latter kind, but range from 
very complex types of models to simple spreadsheet models. For the purpose of consistency the term 
model is therefore used throughout the chapter. 


5.2, OVERVIEW OF MODELS 


This section is a comparison on the different models that are currently in use by the nations taking part in 
the study. The first conclusion that can be drawn from the matrices is that no specific model for a certain 
phase is mandatory for any nation. Many nations do, on the other hand, have some recommendation on 
what type of model that should be used. 


5.2.1 The Different Types of Models Identified 


To begin with, among the models used by the nations, four different types of models have been 
categorised. These are: models for optimisation, simulation, estimation and for decision support. Each is 
briefly described below. 


5.2.1.1 Estimation Models 


This represents a broad spectrum of models that are used at the core of the life cycle costing process. 
As shown later in the chapter, the estimation type of model is often used, in all the PAPS phases. 


' In some occasions it might be helpful to define a so called LCC model framework including the use of several models 
operating together to complete the LCC framework (see Sub-section 2.7.6). 
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Estimation models are all types of models dealing directly with the estimation and calculation of cost. 
The estimation of cost can, in turn, be supported by some other type of method, but in the case of the 
estimation model the main objective is to come to some sort of conclusion as to the level of cost for a 
system or sub-system. 


Since this is a wide model category there are many examples of the estimation type models being used, 
both in terms of commercially available and those developed in-house. One common feature often found 
is that all the models employ a defined cost breakdown structure. These models are often tailored to a 
specific programme and, for those developed in-house are often implemented in a spreadsheet 
environment. 


5.2.1.2. Decision Support Models 


In this category, many types of operational research models with the purpose of choosing or ranking 
between different alternatives can be found. The models are typically based on soft management science 
approaches such as analytical hierarchical process or on multi-criteria decision analysis techniques. 


5.2.1.3 Simulation Models 


This category contains all the models based on one of the simulation methodologies outlined in Chapter 4. 
This therefore includes models using system dynamics and discrete event simulation. In addition, models 
using Monte Carlo simulation have also been included. 


5.2.1.4 Optimisation Models 


This category contains all the models that are based on some type of optimisation method, be it mathematical 
programming, heuristics, or other types of optimisation approaches. These models are most frequently used 
as support methods for the life cycle cost estimation process. For example, they are frequently employed to 
determine stock levels, maintenance regimes and supply chain impacts. 


5.2.2 The Models Used by the Nations — General Overview 


A first look at the different types of models shows that all nations use some form of in-house developed 
model for LCC analysis. The majority also use some type of commercial model. However, some nations 
do not use any type of commercial model; instead they rely entirely on in-house developed models and/or 
other types of methods such as expert analysis. 


Many different types of models are used by the nations. There are for instance hundreds of models, 
the bulk of which are in house developed, but for this study, nearly 40 different models have been 
identified in the matrices. Nearly half of them are commercially available models and the other half have 
been developed in-house. There are differences between the in-house developed models, but one common 
feature is that many are developed in a spreadsheet environment and often tailored for each specific 
programme or project. The in-house developed models are, generally, only used by the nation that has 
developed the model. 


Of the four categories of models identified the most prominent is that of estimation. This statement is still 
valid in considering the use of commercial or in-house models. 


Based on the sample size of 40 models, Figure 5-1 shows the number of nations that are using a certain 
type of model in each PAPS phase. It also gives an indication of which types of models that are used most 
frequently among the nations. Note that the matrix indicates the number of nations and not the number of 
models. 
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Figure 5-1: Analysis of Nations’ Matrices. 


For life cycle cost analysis all the nations have in-house models that are based on a defined CBS 
(cost breakdown structure). Data for these models can be estimated by empiric methods and or by 
parametric formulae (sometimes both methods are used for completeness). For more complex cost 
relationships some nations use other model outputs as an input to the overall CBS model. One example of 
this is to use a model for optimisation of spares to calculate the cost of stock inventory (both in terms of 
initial investment and annual cost of spares). This provides a cost-effective support cost baseline. 


During the early PAPS phases most of the cost models are used to support operational analysis studies 
therefore implying the need for high-level analysis and the use of in-house developed models is common. 


In the later PAPS phases most models are used to support investment appraisal, logistic modelling and 
through life management planning. The quantity and range of applications of the models are greater from 
the project definition phase than for the earlier phases, this is valid both for in-house developed and 
commercial models. 


Only a few nations use optimisation or simulation models. These are mainly used from the project 
definition phase and onward for optimisation of logistics resources and simulation of the support system. 
Optimisation and simulation methods are used to analyse system availability and endurance to support 
operational and tactical planning of technical system in a cost-effective way. 


5.2.3, Models Used in Each Phase 


This sub-section summarises on what types of models are being used in each PAPS phase by the nations. 
There is a strong similarity between the phases, but for the purpose of completeness each PAPS phase is 
described below. 


5.2.3.1 Mission Need Evaluation Phase 


In this phase most of the nations use models for estimation. Only a few nations have stated that other types 
of models are used. The most frequently used of the other type of model are those for decision support. 
There are only three nations that state the use of commercially available models for estimation, whereas 
the in-house developed models are used in greater numbers among many more nations. 
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One reason that nations mainly use in-house developed models in this phase could be that the level of 
detail and availability of data is poor in this phase. This forces the cost estimator to resolve to construct 
their own model. The analyses made in this early phase are often not made by experienced cost estimators 
and the reason for this is since the estimates by default are very rough, the perceived benefit of the 
experienced estimators might be negative. However, it is recommended that a cost estimate is made by 
suitably experienced personnel. 


5.2.3.2 Pre-Feasibility Phase 


The same pattern as in the previous phase can be observed. There is still a large collection of models used 
for estimation purposes and among those; the in-house developed ones are employed in greater numbers. 
This is probably even more so than is shown by the responses given by the nations, since five nations 
report the use of spreadsheet models. There are probably several of these spreadsheet models in use in 
each of these nations. There are, however, a few more commercially available ones being mentioned here 
and more than one nation is using them. One nation reports the use of more than one model for estimation 
purposes, therefore there seems to be an evolving acceptance of the commercially available models. 
The use of models for decision support has dropped to just one nation using an in-house model. 


The strong preference for in-house developed models is probably, as already stated, a result of poor data 
availability. In this phase however, more experienced cost estimators are involved in the process, 
reflecting the growing use of commercial models. 


5.2.3.3 Feasibility Phase 


Again, the same pattern and trend as in the previous phase can be observed. The bulk of models used by 
the nations are still for estimation purposes and the bulk of these are in-house developed ones. In the use 
of commercially available models there seems to be a preference for parametric estimating models. 
The reported use of models has ended. 


The reasons for the continued trend for in-house developed estimation models are believed to be the same 
as in the previous phase. 


5.2.3.4 Project Definition Phase 


The overall picture is the same in this phase with the bulk of models used for estimation purposes. In this 
phase more nations use a couple of more commercially available models. And more nations use a few 
different models for their estimation purposes. Of the models that the nations have reported roughly a third 
are commercially available ones. This is a slight misrepresentation of the fact that, as mentioned before, 
in the nations that report the use of spreadsheet modelling, this most probably means that more than one 
model is in use within each nation. Of the commercially available models for estimation, models dealing 
with parametric estimation seem to be preferred. 


A slight trend for a growing use of models for optimisation and simulation can be seen in this phase. 
These models are most probably used in support for estimation models and the growing use is most likely 
a consequence of more data being available. 


5.2.3.5 Design and Development Phase 


As was mentioned under Section 5.2 the overall picture is the same in all the phases, with a strong 
preference in all nations for models doing estimation. In this phase, the trend mentioned for the project 
definition phase, with a wider spread use of models for optimisation or simulation is still in place. 
These optimisation models are either used to optimise stocks of spares and use this as input in the 
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calculation of a larger investment programme, or they are used to estimate cost of optimised inventory of 
spares. 


The fact that a growing portion of all models reported are commercial indicates a growing reliance on 
commercial models and this might be due to the fact that data is more readily available at this and the 
following stages. 


5.2.3.6 Production Phase 


As before, models for estimation are still the ones being used the most among the nations, but the trend 
towards a more widespread use of optimisation models or simulation models or a combination of them, 
continues, and is even slightly emphasised — though it can hardly be said to be widespread. There are 
almost as many commercial models as in-house models reported in this phase. Again, this appearance 
might be false, since the use of Excel-based models might indicate that more than one such model is used 
in each of the nations that have reported their use. 


5.2.3.7 In-Service Phase 


The picture is almost identical as in the production phase, but only three nations are still using commercial 
ones. As in the three previous phases, a growing number of nations are using optimisation models or 
simulation models. The distribution between the use of in-house developed models for estimation and 
commercial ones is about the same as in the production phase. 


5.2.3.8 Disengagement Phase 


Regarding models for estimation, there is a slight drop in numbers in both in-house and commercial 
models, but the number of nations employing them is the same. The number of models used for simulation 
is the same as in the four previous phases, but only one nation still uses a specific model (commercial) 
for optimisation. In this phase, one other nation again reports the use of a decision support model. 


5.3. RECOMMENDATIONS 


From the material, it is clear that what has here been labelled models for estimation are the preferred 
models for life cycle cost analysis. There is also a clear pattern that to supplement models for estimation, 
more models for decision support are used in the earliest phases and more models for optimisation and 
simulation are used later on. 


To ensure Best Practice, the use of more than one model is recommended. For the purpose of verifying 
results, more than one method of estimation should be used and this means that if data permits, two or 
more models for estimation can be used. If data is available, the use of models for simulation and/or 
optimisation to supplement models for estimation is recommended. The use of multiple models should 
always be evaluated on from a cost-benefit point of view. In the earliest phases, scarcity of data often 
means that the recommended model to use is a decision support model. 


When it comes to multi-national programmes, there is a need for all participating nations to understand 
and trust the models used. The participating nations will need to agree on at least one common model or 
framework (see Sub-section 2.7.6). There is also a need to be clear on which data to use and how to collect 
this data. One way of handling these needs is to use one or more commercial models, maybe supplemented 
with national models, for multi-national programmes. 


There is no single model, of all the models presented by the nations that cover all aspects, so in Section 5.4 
there will be a description of some of the more prominent attributes a good model should have. 
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These attributes can also be used as a checklist when creating or constructing an in-house model or when 
evaluating a commercially available model. 


5.4 DESIRED MODEL ATTRIBUTES 


The characteristics” of a high quality cost estimates are: 


Accuracy; 
Comprehensiveness; 

Ability to Replicate and Audit; 
Traceability; 

Credibility; and 


Timeliness. 


Each of the above should demonstrate these quality characteristics in the following ways: 


Accuracy — Cost estimating relationships (CERs) will be the result of regression analysis with 
good curve fits and minimal error bands, making them valid predictors of cost. Estimates should 
be unbiased, not ‘low balled’ or overly conservative, but based on an assessment of the most 
likely costs. Underlying data will have been correctly normalised for technical baseline and for 
inflation using appropriate guidance. The time phasing of the estimate should also be logical and 
accurate. 


Comprehensiveness — Estimates should use a cost breakdown structure that is at a level of detail 
appropriate to ensure that cost elements are neither, omitted or double-counted. All the cost 
driving ground rules and assumptions must be detailed in the documentation of the cost estimate. 


Replicability and Auditability — The estimate should be presented in a cost breakdown structure 
and work breakdown structure that is fully traceable to the system specification. The estimate 
documentation should include source data, significance and goodness of fit statistics for CERs, 
clearly detailed calculations and results and explanations for why a particular method or reference 
was chosen. An independent reviewer must be able to follow the estimating process, repeat the 
calculations and arrive at the same answer. 


Traceability — Data should be traceable back to the source documentation. 


Without these characteristics the estimate will not be credible, which is the most important quality of a 
good estimate and the benefits just discussed will be much harder to realise. Finally, an estimate must be 
timely. The best estimate in the world does no good if it is too late to provide decision makers the insight 


needed. 


A cost model must therefore be able to demonstrate that it meets the characteristics listed above and is 
fully documented in order to justify the life cycle cost estimate produced. 


° These characteristics are given in the Society of Cost Estimating and Analysis training programme — CostProf © — 
Programmed Review of Fundamentals. They are replicated here by kind permission of the SCEA Office. 
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5.5 COST MODEL VALIDATION - A PRACTICAL PROCEDURE 


5.5.1 Preamble 


The following sub-sections set out a practical procedure for validation’ which can be carried out 
irrespective of the amount of detailed data on specific projects available. However, the confidence which 
can be placed on the validation is obviously dependent on the number of projects for which outturn costs 
can be provided and the reliability of those outturn costs. 


5.5.2. The Procedure 


The procedure described here would be applicable to the form of model comprising only cost estimating 
relationships derived by statistical analysis of the costs of the past projects and relating costs to objective 
quantitative design and/or performance characteristics of the product. It can be used also for methods 
involving non-physical ‘complexity factors’ or equivalents providing that those factors are all derived via 
relationships linking to objective quantitative design and/or performance characteristics in unambiguous 
and repeatable fashion. In that case, those relationships must be declared prior to any trials and adhered to 
without change throughout such trials. However, it is important to note that there is a numerically 
important class of model that relies upon the subjective judgement of the user to set values for ‘complexity 
factors’ or equivalents. For such models, validation is neither possible by the means described here nor by 
any other method. 


5.5.3. Review of Basic Data 
Obtain a list of projects upon whose costs the model being validated is based, including names of projects. 


The important thing at this stage is to establish that the projects on which the model is based all belong to 
a well defined class or generic type. A list is sufficient for this purpose. 


5.5.4 Test Regression 


Predicted costs are then used as the explanatory variable in a test regression to derive: actual cost = 
f(predicted cost). The test statistics (n: number of observations, r: correlation coefficient , o: standard 
error) of such a regression provide a summary of the performance of the model. 


5.5.5 Calibration Tests 


The model is tested against data not used in its construction, using outturn values of design and performance 
characteristics as input. 


5.5.6 Prediction Tests 


A more rigorous test than calibration is the prediction test. In this test, outturn cost data not used in the 
construction of the model is compared to the estimates made by the model based on contemporary 
accounts of the performance and/or other characteristics required of the project at an earlier stage of its 
evolution (typically when development of the project in question was commenced). 


As with calibration tests, the mean and population standard deviation of the cost estimating errors are 
calculated. It is to be expected that errors incurred during use in the prediction mode using performance 


> Reference is made here to the procedure for validating cost estimating models used in cost forecasting. The full statistical and 
practical procedure reference PV/11/098 is available from HVR Consulting Services Ltd. 
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requirements will be larger than in the calibration mode when working from details of the design of the 
equipment in question. However, it may be difficult to find sufficient data to carry out an adequate 
prediction test since people constructing cost models tend to base their models on all the projects on which 
they have outturn data. They are naturally reluctant to set data aside for use in prediction tests when they 
could be incorporated in the model to its benefit. 


5.5.7 Mixed Tests 


In some cases, the actual projects on which the model is based may not be known and in this case the 
person carrying out the validation must depend on his own database. In this situation, he simply compares 
the outturn costs of as many projects as possible of the specified type with the costs estimated using the 
model and calculates the mean and population standard deviation of the estimating error. It is important, 
however, that the database used by the person carrying out the validation is agreed prior to the trial. 


5.5.8 Frequency/Probability Distributions 


For some models, there may be sufficient data to make possible the construction of graphs showing the 
probability of forecasting errors exceeding a given percentage of the actual outturn along the lines of 
Figure 5-2 (or as histograms of the frequency with which forecasting errors occur). Such graphs are 
exceedingly useful, in that they reveal both the accuracy of estimates and any bias. 


Estima = Outturn 


Probability 
ow 
o 
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Figure 5-2: Comparison of Two Models. 
However, a very large data set is required for meaningful results to be obtained. 


5.5.9 Judging the Comparative Validity of Models 


The procedure described above can enable those responsible for validating models, first to establish 
qualitatively that each model covers a well defined class of equipment and then calculate a number of 
measures of its comparative validity. The word ‘comparative’ needs to be emphasised. The procedure is 
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best used to judge whether one model is more likely than another to be useful, having regard to the fact 
that its application to future projects will invariably involve some extrapolation beyond the data set on 


which it was based. However, it is necessary to take complete account of the prediction errors set out in 
the statistical methods upon which the cost model is based. 
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Chapter 6 - DATA COLLECTION 


In terms of time, effort, and resources consumed, collection of data is a major part of a life cycle cost 
study. Life cycle costing is a data driven process, as the amount, quality and other characteristics of the 
available data often define what methods and models can be applied, what analyses can be performed, and 
hence, the results that can be achieved. 


It is important to bear in mind that, because the quality and value of life cycle cost and life cycle 
management analyses is highly dependent on the quality of the available data, good data represent real 
value for a materiel programme. Conversely, real costs are associated with the collection and storage of 
data. 


This chapter will take the broad view and define data as any type of information used in a life cycle cost 
analysis. 


Data collection for risk analysis is a topic that requires extra attention, and is therefore covered separately 
in Sub-section 7.4. 


6.1 EVOLUTION THROUGH THE LIFE CYCLE 


As a system progresses through the life cycle, the types of data available evolve in a number of ways. 
As this in turn defines the task of data collection and the life cycle costing process in general, 
it is important to be conscious of these developments. 


First and foremost, the amount of data available increases as the system becomes better defined. 
Obviously, very little is known about the end system when a project begins and all that exists is an 
identified capability gap or a general concept, whereas, when a system is in service, the system and its 
environment can be documented in almost infinite detail. Unfortunately, because uncertainty, risks, 
and opportunities decrease as the life cycle progresses, the need for knowledge is the greatest at the 
earliest stages. This means that more time and resources should be allocated to the data collection effort at 
the earlier stages of the life cycle. 


The characteristics of the available data will change as well as the progress of the life cycle. Early data 
will tend to be softer and be in a more aggregated form, because hard numbers and detailed information 
are not yet available. For example, an early concept may stipulate a high engine reliability, which is later 
quantified as an expected or required mean time between failures for the engine itself and later still for the 
various parts of the engine. 


Chapter 3 gave an overview of the relevant data sources for each phase of the life cycle of a system. 
It is recommended to anticipate future data requirements and to collect data accordingly. 


6.2 DATA SOURCES 


Life cycle costing requires a wide variety of data, and these must be collected from an even wider variety 
of sources. A distinction can be made between primary data, coming directly from the source, 
and therefore generally of better quality and utility, and secondary data, which is derived, possibly altered 
in the process, and therefore of inferior value. 


When preparing a cost estimate, estimators should consider all credible data sources. However, primary 
sources of data should be given the highest priority for use whenever feasible. 
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Table 6-1 illustrates the difference between primary and secondary data, using the following definitions of 
primary and secondary sources of data for classification purposes: 


e Primary data are obtained from the original source. Primary data are considered the best in 
quality, and ultimately the most reliable. 


¢ Secondary data are derived (possibly sanitised) from primary data, and are therefore, not obtained 
directly from the source. Because secondary data are derived (actually changed) from the original 
data, it may be of lower overall quality and usefulness. 


Table 6-1: Sources of Data 


Sources of Data 


Typical Data Sources Source Type 
Basic Accounting Records 
Either (Primary or Secondary 


Historical Databases Either 


As already indicated, in the early stages the system itself is non-existent, so data will have to come from 
comparable systems and programmes. This means that data will primarily come from outside the 
programme, whereas in the later stages, more data will be generated internally by the project itself. 
This makes data sources for the earlier phases harder to identify and access and it makes data collection 
and validation more difficult. 


Outside data sources can be industry or other branches of the military or the government. They can be 
domestic or from other nations. It is worth noting the existence of the NATO agreement on the 
communication of technical information for defence purposes (Reference: [67]), which covers the 
communication of proprietary technical information among the signatory NATO nations and NATO 
bodies. 


In these early stages, a certain creativity and flexibility will often be necessary, because most data will 
have to come from outside the programme. This means that comparable systems and programmes, as well 
as the relevant data sources within these, must be identified. Data from comparable systems must then be 
modified based on differences between systems with respect to performance, complexity, maturity of 
technology, etc. (see Sub-section 6.5 on data normalisation). This task is highly dependent on the specifics 
of the individual programme, and no generalised or automated methods can be applied. Furthermore, 
care must be taken to ensure that reference systems are in fact comparable to the system of interest. Use of 
radically different technology, differences in operating profile, etc., for a new system may mean that data 
from older systems are irrelevant. 


Ideally, it should be proven statistically that a credible relationship exits between the relevant data for a 
reference system and the system of interest. This, however, is rarely feasible in practice, but since the use 
of assumed or uncertain relationships should be reflected in the uncertainty of the final estimate, it is 
recommended that evaluation, statistical or otherwise, of the certainty of such relationships are 
performed. 
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At the earliest phases, expert judgment or opinion may be the only available source of information, and it 
represents the extreme aggregate and soft end of the spectrum. Expert judgment can be viewed as data or 
as a source of data, but in essence it is a black box analysis of more or less well documented data and 
experiences. Since it is largely impossible to validate, it is useful if at all possible to get more than one 
opinion. 


Again, these tendencies point to a more manual, creative, labour intensive effort in the early stages of the 
life cycle. 


From the production phase onwards, hard data from tests and in service use, generated within the 
programme itself, begin to become available. Such data are increasingly collected in ERP (Enterprise 
Resource Planning) systems and other database systems such as the VAMOSC (Visibility And 
Management of Operation and Support Costs) databases maintained by various branches of the 
US military. This in turn allows more automated collection of data in reports and tables for use in the 
further study of the system itself or of comparable systems at an earlier stage in their life cycle. However, 
because the need for and availability of various types of data changes throughout the life cycle, 
data collection is and remains in essence a manual task. 


Some data sources, such as contractors and sources external to the defence organisation, will change often, 
maybe with each new life cycle cost programme, while others, such as maintenance and accounting 
functions within the defence organisation, will be used repeatedly by the life cycle cost function. 
It is immensely helpful if there is an understanding of life cycle cost by these repeat providers of data, 
including how and why it is done and how it is used. This will help ensure that the cost analyst is given the 
right data of the best quality. Conversely, if there is a widespread feeling that data collection is a pointless 
chore that generates no value, data quality will drop. This will in turn generate poor estimates, 
perpetuating a vicious cycle by lending further credence to negative feelings about life cycle costing. 
Hence, time and effort spent on selling life cycle costing within the defence organisation, informing data 
providers on how their output is used, may prove very well spent. 


It is important in this context to note the value of storing data from past projects and programmes for 
future use. Though it is important to remember the costs associated with storing data, this just underlines 
the benefits of anticipating future data needs, and to store the right data in an appropriate format. 


6.3 DATA DOCUMENTATION, FORMATS AND STANDARDS 


Electronic exchange of data between ERP systems and databases can be a cumbersome and time 
consuming affair if data formats and data models differ. However, a number of standards exist for the 
exchange of life cycle data. For some years, the official NATO standard has been the NPDM 
(NATO Product Data Model), formerly known as NCDM (NATO CALS Data Model). An international 
standard exists in the form of ISO 10303-239 (STEP, Application protocol 239 — Product Life Cycle 
Support, PLCS), which has been put forward by NATO to be adopted as STANAG 4661’. 


PLCS provides an application specific but generic and flexible data model for life cycle data. Industry and 
organisations can tailor this for their specific application using RDL (Reference Data Libraries). Needless 
to say, this generic data model for all life cycle data is extremely large and complex. This problem is dealt 
with by defining DEX (Data EXchange Sets), which are subsets of the data model suited for a particular 
business process, such as the DEX D005 on Maintenance Plan. 


Future development of PLCS, including the definition of new DEX, is overseen by a technical committee 
under the OASIS (Organisation for the Advancement of Structured Information Standards) consortium. 


' PFP(AC_327)D(2006)0002 (Draft Edition 1), STANAG 4661 on Product Life Cycle Support. 
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The Cooperation on Defence Implementations of PLCS (CDIP) has been formed to undertake defence 
related aspects of this work. 


As indicated, the PLCS is a very large and technically complex mechanism, and implementing PLCS 
would be a huge undertaking for any organisation. It is therefore not feasible, for instance, to require 
industry or other partners to adopt PLCS as a pre-condition for collaboration. In such cases, other, simpler, 
standards must be found (The OASIS homepage is a good source). Alternatively, ad hoc solutions in the 
form of agreed upon and documented templates, etc., may be used, but this makes the data harder to use at 
a later date for other projects or purposes. When possible, officially defined and accepted standards should 
be preferred. 


In the long term, as it has been adopted as an ISO standard and by NATO as a STANAG, and with 
continued support and development by industry and nations, PLCS has potential to be an important tool to 
help collect and exchange high quality, well documented data. 


6.4 DATA FROM SUPPLIERS/CONTRACTORS 


Various mechanisms can be employed to gather data from contractors. The use of a life cycle cost 
questionnaire is particularly recommended and examples of such questionnaires are provided at Annex C. 


Though today data can be shared easier than ever before, it takes time and resources to measure, collect, 
and manage data. Therefore, it must be understood that data comes at a price, and that it is entirely 
reasonable for contractors, suppliers, and others to charge a price for data, even if this is part of a major 
weapons system purchase. The upside of this is that it is then possible and reasonable to make demands on 
the validity and accuracy of the data received. 


In this context, it is particularly important to anticipate future data requirements and to frame agreements 
and contracts accordingly. The timing of collection and delivery of the data as well as the contents and 
formats of the required data should be clearly defined. The quality and reliability of data from suppliers is 
often inferior. Whether this is caused by a lack of incentive or ability, it is recommended to have 
previously agreed upon and well documented templates or standards for the data to be exchanged. 


However, a contractor will often have an interest in presenting a system in the best possible light. 
Care must therefore be taken to secure that the data received from contractors, or other sources with a 
vested interest in a programme, are accurate and unbiased. This can take the form of a contract or other 
binding agreement which puts some form of penalty, possibly some or all of the costs and risks arising 
from errors and omissions, on the contractor. 


The UK MDAL (Master Data and Assumptions List) (see also Sub-section 2.3.2.1; see also Reference 
[73]) is one well documented mechanism for ensuring that all stakeholders buy into a common and clearly 
stated understanding of the project and the system of interest. A document like the MDAL needs to be 
dynamic and iterative, but it also needs to be frozen at certain points (milestones) in the life cycle to 
provide documentation of decisions taken at this point and support audit of these decisions. This process is 
illustrated in Figure 6-1. 
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Define Source & Collect IPTL* 
Outputs and/or Data. Customers 

Options Fill Data gaps uncertainty Approve Issue 0 
for this issue of with agreed against Data MDAL 

MDAL Assumptions Assumptions (copy to archive) 


Stakeholders review Data & 
Assumptions regularl 


Figure 6-1: Development of Master Data and Assumptions List”. 


A similar function is fulfilled for major acquisitions of the US DoD by the CARD (Cost Analysis 
Requirements Description) (further details are provided at Section 2.3.2.2). The CARD is used to formally 
describe an acquisition programme, including the system itself, for the purpose of preparing all cost 
estimates for the programme. It is provided at major milestone decision points for materiel programmes 
and for major information system programmes whenever an economic analysis is required. 


Furthermore, the US DoD has developed the CCDR (Contractor Cost Data Reporting) system for 
accumulating actual contractor costs necessary to analyse costs efficiently and effectively. The CCDR 
contains forms and templates for reporting required cost data and provides extensive guidance on 
mandatory and recommended policies and processes to be followed by contractors for major materiel 
systems. The main components of the CCDR system are: 


¢ The cost and software data reporting plan. This form, referred to as the CSDR plan, specifies the 
WBS? (work breakdown structure) elements, the specific report format and the reporting 
frequency. 


¢ The cost data summary report. This form captures all contract WBS elements at the level 
specified in the CSDR plan and includes both recurring and nonrecurring breakouts. 


¢ Functional cost-hour and progress curve report. Part I of this form, functional cost-hour report, 
is directed at selected WBS elements where more detailed cost data are needed. It contains a 
functional breakout (e.g. engineering and manufacturing) and a cost element breakout (e.g. direct 
labour and material) within functional categories. Part II, progress curve report, captures recurring 
costs on lot or unit data for selected WBS elements. 


A similar mechanism exists for information systems and other software heavy acquisitions in the form of 
the SRDR (software resources data report). The CCDR and the SRDR are each described in a separate 
manual which are available on the internet: 


¢ DoD 5000.4M-1 Contractor Cost Data Reporting (CCDR) Manual, 4/1999 (See: Ref. 78) 
http://www.dtic.mil/whs/directives/corres/html/50004m1.htm 


« DOD 5000.4-M-2 Software Resources Data Report (SRDR) Manual 2/2004 (See: Ref. 79) 
http://dcarc.pae.osd.mil/srdr/DOD50004M2.pdf 


> From www.ams.mod.uk/ams/content/docs/wlc/wlcmdal.htm 


3 WBS is similar to Cost Breakdown Structure (CBS) as defined by SAS-028. 
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While data collection through formal reports such as CCDRs and SRDRs is extremely important and 
beneficial, there is still no substitute for taking the time to understand and verify the accuracy of historical 
information, and the programmatic context in which it was obtained. 


6.5 DATA NORMALISATION 


In Sub-section 6.2, the concept of primary and secondary data was introduced, and it was indicated that 
primary data are preferable. However, since raw data come from a variety of sources, there is generally a 
lack of uniformity in data and therefore a certain amount of normalisation will be unavoidable. Generally 
speaking, data normalisation covers changes and adaptations made to primary data to make it applicable in 
a given model. It is defined by SCEA (Society of Cost Estimating and Analysis) as: 


¢ To adjust a measured parameter to a value acceptable to an instrument or technique of 
measurement. 


¢ For a data base: to render constant or to adjust for known differences. 
¢ For cost or dollars/Euros: Then-Year dollars/Euros and/or actuals are escalated to a common Base 
Year for comparison. 
The last definition in particular will be relevant for cost data. However, normalisation can take many 
different forms and have different specific purposes, such as: 


e Adjusting costs to a common year or adjusting to different inflation or discounting mechanisms or 
other variations in accounting standards. 


e Adjusting system or parts costs for technical specifications like size, weight, complexity, 
technological maturity, etc. 


e Adjusting costs or technical performance data such as failure rates for different operating profiles 
like operating temperature, mileage, etc. 


e Adjusting prices for lot size, learning curve considerations, producer capability and maturity, etc. 
e Adding cost items not originally included, for example through error or because of a different 


costing scope, or removing cost items which are not applicable. 


Regardless of how data are normalised, exact, complete and detailed documentation of the process is very 
important. This is the case whether normalisation of primary data is performed as part of the life cycle cost 
estimating process or secondary data has been obtained for use in life cycle cost estimation. Serious errors 
can occur if data is not properly understood and interpreted. It is therefore vital to fully understand data 
and to know where data is coming from. 


The next sub-sections describe the relevant issues related to cost data normalisation. 


6.5.1 Base Year 


The first step is to establish an appropriate base year for data normalisation. A base year is a fiscal year 
whose mid-point is selected as a reference point for computing an index. A programme base year is 
usually the year of initial programme funding. Normalising to the programme base year facilitates the 
analysis of data on a comparative basis during the cost estimating process. 


6.5.2 Constant Years versus Current Years 
An estimate is said to be in constant dollars or Euros if costs are adjusted so that they reflect the level of 


prices expressed in the dollars/Euros of a fixed base year. The terms real or constant are used 
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interchangeably to refer to the purchasing power of the dollar/Euro for the specified base year. When cost 
estimates are stated in real dollars/Euros, the implicit condition is that the purchasing power of the 
dollar/Euro will remain unchanged over the time period of the programme being costed. Normalising data 
due to inflation allows an estimator to track price changes explained by other causes. 


Current year costs reflect the purchasing power in existence when expenditures are actually made. 
Prior costs expressed in current year dollars/Euros are the actual amounts paid out in those years. Future 
costs stated in current year dollars/Euros are projected amounts to be paid, including the changes in the 
purchasing power of the dollar/Euro. Terms as current, then-year and nominal dollars/Euro are sometimes 
used interchangeably. 


Cost estimates normally are prepared in constant dollars/Euros to eliminate the distortion that would 
otherwise be caused by price-level changes. This requires the transformation of historical or actual cost 
data into constant dollars/Euros. For budgeting purposes, however, the estimate must be expressed in 
current year dollars/Euros to reflect the programme’s projected annual costs by budget appropriation. 
These annual appropriations actually are expended over a number of years. This requires that the 
appropriation request takes into account the effect of the anticipated inflation that corresponds to the 
outlay pattern for each appropriation. The dilemma facing the estimator is how to bridge the gap between 
the estimate in constant year dollars/Euro and a budget request in current year dollars. 


6.5.3. Using Indices 


As mentioned before in order to compare cost incurred or estimated for different years asks for converting 
costs from one year to reflect the price level of another year. Price or other indices can be used to 
accomplish this conversion. An overview is given (see reference: [79]) on how to select the right index 
and how to use these indices. 


6.5.4 Exchange Rates 


The use of foreign exchange rates is a problem unique to analyses performed on international programmes 
where costs are stated in foreign currencies. This is particularly prevalent in multi-national programmes. 
It is usually difficult to obtain reliable forecasts of foreign exchange rates. One approach is to assume that 
if inflation in the foreign country is greater than own country inflation, the rise in foreign prices will be 
fully offset by currency devaluation (this is the concept of ‘purchasing power parity’). 


In multi-national programmes, usually a base currency and a base year date is agreed upon, e.g. it is 
agreed upon that all costs will be express in constant Euros, July 1“, 2006 or economic conditions July 1°, 
2006. In this case costs calculated in other currencies need to be expressed in Euros using the exchange 
rate for that particular date. For example, 1 US dollar = 0.83 Euro. This exchange rate would then be used 
throughout the study. 


A typical process for using exchange rates is shown below: 


¢ Step 1. If the foreign values are expressed in constant value, note the base year. If they are first 
expressed in current value, deflate by using the appropriate foreign compound index. The result of 
this step is that the costs will be expressed in constant currency for a known base year. 


¢ Step 2. Multiply the result from step (1) by the own/foreign currency exchange rate for the known 
base year. The result of this step is the constant currency costs. 


e Step 3. With the costs now established, multiply these costs by the proper inflation values using 
the base year established at step (1). These will be the costs to be included in the estimate. 
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6.5.5 


Inflation 


Inflation is a consistent rise in the costs (prices) of goods and services over time. To introduce the effect of 
inflation into economic analysis, the following inflation-related terms are defined: 


6.6 
[73] 


[74] 


[75] 


[76] 
[77] 
[78] 
[79] 
[80] 
[81] 
[82] 
[83] 


[84] 


Constant currency (Euros, Dollars, GB Pounds, etc.). Constant year currency values are the result 
of having the effects of inflation removed. Constant year currency values are always associated 
with a base year; for example, fiscal year (FY) 1995. 


An estimate is in constant currency if the costs for all the work is adjusted so that they reflect the 
base year level of prices. When prior or future costs are in constant currency, the figures given are 
adjusted to presume that the “buying power” of the currency was the same and will continue to 
remain the same as in the base year. The use of constant currency assists in the evaluation of 
resource requirements over time because it removes distortions which are attributable only to 
price level changes. With the removal of inflation, the true cost growth of a system can be more 
readily determined. 


Current or then year currency. Current year values are expressed in the value of the year in which 
a cost is expected to occur, and therefore reflect the effects of inflation. The term “current year” 
means that the amount is appropriate for the year in which the money is expected to be expended. 
When prior costs are stated in current year values, these values are the actual amounts paid out. 
When future costs are stated in current values, the figures given are the actual amounts which will 
be obligated including any amount estimated for future price change. When making estimates for 
the future, assume a base buying power for each currency unit (constant values) and then apply an 
inflation factor that converts the estimate into current year values. 
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Chapter 7 - UNCERTAINTY AND RISK 


7.1 INTRODUCTION 


Life cycle cost estimates of weapon system acquisition programmes are inherently uncertain and risky. 
Estimates are often made when only 5% of a programme’s total cost is known. Years of system 
development and production, and decades of operating and support costs, need to be estimated. Estimates, 
in turn, are based on historical samples of data that are almost always messy, of limited size, and difficult 
and costly to obtain. Indeed, great efforts are usually required to squeeze usable information from a 
limited, inconsistent set of data. And no matter what estimation tool or method is used, historical 
observations never perfectly fit a smooth line or surface, but instead fall above and below an estimated 
value. To complicate matters, the weapon system under study is often of sketchy design. Only limited 
programmatic information may be available on such key parameters as schedule, quantity of units to be of 
the system may actually change as the system proceeds through development and even production. 
Increases in system weight, complexity, and lines of code are commonplace. 


For all of these reasons, a life cycle cost estimate, when expressed as a single number, is merely one 
outcome or observation in a probability distribution of costs. That is, the estimate is stochastic rather than 
deterministic, with uncertainty and risk determining the shape and variance of the distribution. To better 
support senior leadership, some sense of risk and uncertainty needs to be presented along with the point 
estimate. This chapter, through the following sections, demonstrates how to do this by providing guidance 
on the following: 


¢ Definitions. 

¢ General approach. 

¢ Risk data collection. 

e Estimation process. 

¢ Cost and budget Risk. 
¢ Sensitivity analysis. 


e Appendix 1: Optimism Bias. 


7.2 DEFINITIONS 


The terms “risk” and “uncertainty” are often used interchangeably, but they are not the same. 


¢ Uncertainty is the indefiniteness or variance of an event. It captures the phenomenon of 
observations, favourable or unfavourable, falling to the left and right of a mean or median value. 


* — Risk is exposure to loss. Or, in a weapon-system acquisition context, it is a measure of the potential 
inability to achieve overall programme objectives within defined cost, schedule, and technical 
constraints, and has two components: (1) the probability/likelihood of failing to achieve a particular 
outcome, and (2) the consequences/impacts of failing to achieve that outcome. ' 


Risk and uncertainty, then, are related. Uncertainty is probability while risk is probability and 
consequence, as Figure 7-1 shows. The next section of this chapter outlines a procedure for estimating risk 
and uncertainty. 


' “Risk Management Guide for DoD Acquisition,” Fifth Edition, June 2003; U.S. Defense Acquisition University; p. 7. 
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Probability of Occurrence 
(Uncertainty) 


0.8 


0.2 


Severity of Consequence 1+—————_» 


Figure 7-1: Risk Matrix. 


7.3. GENERAL APPROACH 


There are a wide variety of methods and models available for conducting risk and uncertainty analysis of 
life cycle cost estimates of weapons systems. These include simple techniques such as adding a risk factor 
or percentage to a bottom-line estimate (see: Appendix 1, “Optimism Bias’) and sensitivity analysis. 
Each, if used properly, can give scientifically sound results, but first a word of caution — based on the 
collective experience in cost estimating within the government and in the private sector, it is fair to say 
that the sophistication and underlying theory of many popular models often far exceeds the quality of the 
basic data inputs.” 


There is simply no substitute for taking the time and effort to understand the technical risks and challenges 
in developing and producing sophisticated defence systems.” Historical analogies must be obtained. 
Information from subject matter experts must be elicited. Risk and uncertainty analysis cannot be 
relegated to an eleventh hour exercise based on flimsy inputs." 


7.3.1 Overview 


Figure 7-2 presents an overview of a process for estimating risk and uncertainty.° While other techniques 
and variations of this process are available, the paradigm shown below is highly recommended. 


* Informal survey by the Naval Center for Cost Analysis in 2004 of 12 cost-estimating organizations in the United States and 
“Portfolio Management for New Product Development: Results of an Industry Practices Study,” Drs. Cooper, Edgett, 
and Kleinschmidt; Product Development Institute; 2001; page 20. The “popular models” referenced by these authors are 
Crystal Ball and @Risk, the same two commercial models used most frequently in defense cost analysis in the United States. 


: Biery, Hudak, and Gupta refer to this as “the most crucial ... but generally overlooked” step in performing risk and 
uncertainty analysis. “Improving Cost Risk Analyses,” 1994, p. 1. 


‘ Interestingly, Dr. Cooper reports that one firm in his sample studied the historical accuracy of their probability estimates and 
found an average error of 300%. 


> Much of the methodology presented herein was developed by Mr. Tim Anderson, Aerospace Corporation in collaboration 
with Dr. Steve Book MCR Federal Inc. 


7-2 RTO-TR-SAS-054 


UNCERTAINTY AND RISK 


It provides outputs that have proven useful in responding to demands of senior executives for relevant 
information on budget risk on major acquisition programmes. More specifically, the process enables 
decision makers to budget a programme at a specific cumulative percentage level of risk, or, to fully 
understand the consequences of living within an already established budget. And it enables them to know 
the financial impact of specific, discrete risk events such as failure to successfully design a new fighter 
engine within fiscal and schedule constraints. 


Data Collection for Risk 
and Uncertainty Analysis 


! 


Baseline Probability Distribution 


Likelihood 


Modeling Uncertainty 
Monte Carlo Simulation 


vu 


Costs in Millions 


For each stochastic CBS element, choose: 
> Type of probability distribution 


> Parameters of the distribution (e.g., u, 6?) 


> Correlation with other CBS elements 


Risk-Adjusted Probability Distribution 


Likelihood 


Modeling Uncertainty 
Technical Uncertainty & Risk 
Monte Carlo Simulation 


y2 


Costs in Millions 


Figure 7-2: Process of Estimating Risk and Uncertainty. 
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The most important part of the process of estimating risk and uncertainty, and probably the most difficult, 
is data collection and analysis. All variables in the cost estimating model potentially affected by risk and 
uncertainty first need to be identified. These variables often include simple ratios and factors as well as 
more sophisticated CERs (cost estimating relationships) based on regression analysis. Probability 
distributions need to be estimated or selected for each variable. This entails first choosing the type of 
distribution to apply and then estimating the distribution’s parameters such as high, low, and most-likely 
values. Popular distributions for this step include the normal, log-normal, and triangular. There are a 
number of techniques used to cull or estimate distribution types and parameters, such as using checklists 
or sound engineering judgment. Each is described in Sub-section 7.4. 


It is also important in the analysis phase to identify discrete risk events, or unfavourable outcomes that 
might occur in developing, manufacturing, and operating weapons systems. An example might be failure 
of a new, state-of-the-art radar to work as intended when integrated on a ship or aircraft. For each of these 
risk events, probability distributions also need to be estimated or selected as well. 


After the data collection and analysis phase, the analyst first generates a baseline cost estimate using 
Monte Carlo simulation followed by a risk-adjusted cost estimate. The output of each estimate is actually a 
frequency distribution of total costs, or, more technically, a probability density function, rather than a 
single number. It is essential to convey to senior leadership the notion that cost estimates are uncertain, 
that programmes can and do incur difficulties, and that the probability of a cost estimate becoming reality, 
when expressed as a single number, is actually zero. 


In generating the baseline cost estimate, the analyst first regards as fixed the values of the explanatory or 
independent variables (Xs) in each of the cost model’s CERs. Values of the Xs are usually found in the 
programme’s CARD (Cost Analysis Requirements Description), or in the APB (Acquisition Programme 
Baseline). The baseline estimate does capture uncertainty in the relationship between dependent and 
independent variables in each CER. This uncertainty, in turn, results from three possible, though not 
mutually exclusive, sources: 


¢ Limited data. In explaining changes in the cost of any CBS element, the list of relevant factors 
may be extended ad infinitum. However, due to data availability, perhaps only two or three of 
these factors are included in the analysis. Indeed, sometimes it is lucky to get just one relevant 
explanatory variable. The CER, then, becomes an over simplification of the complexities of 
reality. Errors result. 


¢ Human unpredictability. Over and above the total effect of all relevant factors, there is a basic 
and unpredictable element or randomness in human responses that can be adequately 
characterised only by the inclusion of uncertainty in the analysis. This will hold as long as people 
rather than machines acquire and build weapons systems. 


¢ Errors of observation or measurement. Cost and technical data are almost always difficult to 
obtain and are often of less than perfect accuracy. For example, overhead costs from different 
contractors may not be of the same scope or consistency due to differences in ways of doing 
business. Further, even data from the same contractor may differ significantly over time due to 
changes in the company’s accounting system. Again, errors result. 


In generating a risk-adjusted cost estimate, not only is basic CER uncertainty captured, as above, 
but technical risk and uncertainty as well. Unlike before, the Xs in each of the model’s CERs are now 
regarded as stochastic. Technical, acquisition, and cost-estimating risks are now considered. Variables 
affected might include: 


* Quantity of units to be developed or procured. 


¢ Weight of a platform or system. 
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e External parameters such as the price of oil. 

e System-to-platform integration challenges. 

¢ Number of drawings. 

¢ Number of SLOC (source lines of code) or percentage of SLOC reuse. 

¢ Number of test flights. 

e Key schedule milestones such as date of critical design review or date of first flight. 


¢ Cost parameters such as learning curve rates, T1s (first unit costs), and percent award fee, 
assuming these variables are not already covered in uncertainty analysis. 


Further, discrete risk events such as failure to effectively design a new aircraft engine or a new circuit card 
are captured here as well. As before, for each of these risk variables, probability distributions are estimated 
or selected, and Monte Carlo simulation is used to generate a probability density function. 


Finally, the difference in mean values of the two probability distributions, risk-adjusted and baseline, 
represents cost risk. Senior leadership can then set budgets based on how much risk they are willing to 
tolerate. 


The following sub-sections describe the process in more detail. 


7.4 COLLECTING DATA FOR RISK ANALYSIS 


Data is the raw material of risk and uncertainty analysis. It is critical to every estimate. Without good, 
solid data, whether based on historical analogies or on sound engineering understanding of the acquisition 
at hand, the risk and uncertainty estimate will be viewed as merely a guess or an opinion of the cost 
analyst. The more solid the data, the better will be the estimate. 


At the start, it is important to understand the fundamental objectives of the programme, including 
requirements, scope, schedule, technical goals, and evolutionary phases. With this backdrop in mind, 
the following steps should be executed: 


e Identify all potential variables in the cost model affected by risk and uncertainty. It is usually 
helpful to have these in a single input area in a workbook. 


e Identify potential data sources for estimating risk and uncertainty for each of these variables. 
That is, identify various ways of trying to determine a variable’s type of probability distribution 
and its associated parameters such as high, low, and most-likely values. Options include: 


e Using statistical equations from the cost model that give estimated means and variances. 
e Using scorecards, and derivatives thereof. 

¢ Culling ideas and information from subject matter experts. 

e Finding good historical analogies. 


e Identify correlations between stochastic CBS elements. 


e Identify sources for cross-checks. These might include use of alternative methodology, 
comparisons with historical cost growth on similar programmes, and, time permitting, sanity 
checks for completeness and reasonableness with subject matter experts. 


¢ Develop and execute a data collection plan. 


The following sub-sections discuss specific techniques employed in gathering and analysing the 
information required to conduct risk and uncertainty analysis. 
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7.4.1 Techniques 


Data collection techniques are conveniently grouped into these categories, although there is some overlap 
between the three: 


¢ Use of scorecards and their derivatives. 
¢ Use of historical cost data. 


¢ Use of subject matter experts (SMEs). 


7.4.1.1 Scorecards 


Scorecards and their derivatives usually present a vector of numerical values from which SMEs choose to 
assess risk in a given area, such as schedule. Scales, which indicate probability of occurrence or amount of 
risk, might range from 1 to 10 or perhaps from 1 to 3. These numerical values are then translated into 
impact on cost.° 


7.4.1.2 Historical Data 


An excellent source of information on risk and uncertainty is historical data and its use in the cost 
estimation process. Least squares regression analysis, for example, automatically generates an estimate of 
the mean and variance of an equation’s random error term. These values can be input directly, without 
modification, into the Monte Carlo simulation. Further, metrics on growth in key parameters such as lines 
or code or aircraft weight from start of development through initial operational capability are very useful 
in shedding light on possible areas of risk. 


7.4.1.3. Expert Opinion 


Risk analysts have traditionally used historical data as an information source in probability assessments, 
but sometimes the required data is quite difficult to obtain. An important factor, which often causes 
relevant data to be scarce, is that risk analysis typically deals with rare events. Furthermore, the systems 
under study often represent new concepts and technologies for which little or no experience exists. In risk 
analysis where there is little or no relevant historical data, expert judgment is frequently applied. In other 
words, expert judgment is typically appropriate when: 


¢ Data is sparse or difficult to obtain. 
¢ Data is too costly to obtain. 
¢ Data is open to different interpretations, and the results are uncertain. 
e There is a need to perform an initial screening of the problems. 
An expert, in this instance, is a person with special knowledge or skills in a particular area. The selection 


of the expert is important for the accuracy of the results to be obtained. The following criteria may be used 
for the selection process: 


e Experience in performing judgments and making decisions, based on evidence of expertise like 
degrees, research, publications, positions and experience, etc. 


e Availability and willingness to participate. 
e Impartiality. 


e Inherent qualities like self-confidence and adaptability. 


® See Cost Programmed Review of Fundamentals (CostProf), Chapter 9, “Cost Risk Analysis,” Society of Cost Estimating 
and Analysis, 2002, for an important discussion of scorecards, especially the advantages of interval versus ordinal ranking. 
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Further, culling of information from SMEs can be facilitated and made more rigorous by employing some 
of the following tools and procedures. 


¢ Checklists. These can be based on experience of earlier programmes — risk issues can be 
identified and quantified through an examination of what occurred on previous programmes plus 
an overall understanding of the issues that are likely to be problematic on future programmes. 
These issues can be formalised into lists and structured in a way that suits the particular type of 
programme. New programmes can then be examined against the list and an opinion formed about 
each point raised. 


¢ Structured Interviews. These would be held with responsible or knowledgeable staff, perhaps 
using the checklist as the basis. Interviews of this type are best done on a one-to-one basis, free of 
any hint of an inquisition. They can refine the perception of where difficulties may lie as well as 
drawing attention to areas that might not be covered in the checklists. 


¢ Brainstorming Sessions. These are normally held with a group of knowledgeable staff in an 
atmosphere of free speculation and without peer criticism. People are invited to produce as many 
ideas as possible of the risks that might arise. This can lead to a very large number of ideas, some 
of which may be wild speculation. All risk ideas put forward have to be analysed and categorised 
into those that are real and need to be dealt with and those that are largely imagined or are 
extremely unlikely and can be ignored. 


¢ Assumption Analysis. This is where all the basic assumptions are listed and challenged. This is 
often more difficult than it might seem as many assumptions are unspoken and simply never 
considered. They are things that are so familiar that they are taken for granted, even in new 
situations. Assumptions should be tested against both their importance to the programme overall 
and the likelihood that they might prove false. Any that cannot be unreservedly accepted as being 
valid are potential sources of risk and can be treated as such. 


7. ESTIMATION PROCESS 


As mentioned above, each of the factors, ratios, and CERs in a cost-estimating model is usually stochastic. 
The uncertain or random nature of these variables can be expressed as a probability distribution with a 
certain mean and variance. Combining the probability distributions of each of the variables in a large cost 
model for a major weapon system acquisition programme to obtain a total cost probability distribution 
cannot be done mathematically. The number of variables is simply too numerous and the resulting 
calculus too unwieldy. A good, statistically sound alternative is Monte Carlo simulation. 


In this technique, a random sample is taken from the probability distribution associated with each CER 
and each risk variable. Based on the functional form of the factor or CER, arithmetic operations are 
performed to obtain a single estimate of the cost of that CBS element. This is done for each uncertain 
factor and CER in the model. Results are summed into a single estimate of the cost for the entire weapon 
system. This estimate, then, is one observation or experimental result out of an infinite number available. 
The procedure of random number selection and subsequent cost computation is then repeated thousands of 
times to develop a frequency histogram (or probability distribution) of total system cost. 


7.5.1 Baseline Cost Estimate 


In this step, input values such as a SLOC (Source Lines of Code) count, aircraft weight, and radar 
performance are regarded as deterministic. Values of each of these variables are plugged into CERs as 
fixed numbers. Figure 7-3 illustrates the process for one CER corresponding to one cost element. In this 
example, the CER represents the relationship between software development cost and the number of 
source lines of code. The relationship is curvilinear, as are those of many CERs in defence cost analysis. 
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Beyond a certain SLOC count, the software development project becomes so big and complex that the 
probability of success diminishes sharply and costs escalate precipitously. 
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Figure 7-3: Baseline Cost Estimate. 


To generate the baseline cost estimate, a fixed value for SLOC count is input into the CER. For this fixed 
value, there is an entire probability distribution of associated costs.’ The distribution represents cost 
uncertainty. Cost, in this case, is log-normally distributed.* In the Monte Carlo simulation, the computer 
generates a random number for this distribution, based on the value of SLOC. An identical process is 
followed for other CERs (not shown here) in the model. Costs are aggregated for all CBS elements. 
The baseline cost estimate emerges after sufficient repetition of the process. 


7.5.2 Risk-Adjusted Cost Estimate 


The next step is to generate a risk-adjusted cost estimate. 


As Figure 7-4 shows, input parameters such as a SLOC (source lines of code) count are now regarded as 
stochastic rather than deterministic. Hence, distributions for these parameters need to be chosen. These are 
often triangular. Most-likely values, defined by the mode in this case, may now differ from those values 
presented in the CARD (cost analysis requirements description). Once again, a Monte Carlo simulation is 
conducted. First, the computer randomly chooses a SLOC count from the triangular distribution, 
somewhere within the low to high range. These low and high values correspond to median values on the 
population relationship (or curve) between Y and X. For each of these median values, in turn, there is an 


7 This is because the underlying relationship between cost, Y, and SLOC count, X, is stochastic. The relationship is specified by 
the equation Y = aXe®, where o and B are population parameters to be estimated, e is the residual value and ¢ is a normally 
distributed random error term with mean 0 and variance o”;. This error term imparts uncertainty to cost. 


Interestingly, as Goldberger indicates, when a power-function form is used for a CER, attention shifts, “... apparently 
unwittingly, from the mean to the median as a measure of central tendency.” That is, plugging a value of X into the CER 
yields an estimate of the median value of Y rather than the mean. This is a little known fact in econometric modelling and in 
defense cost analysis. See Goldberger, Arthur S., “The Interpretation and Estimation of Cobb-Douglas Functions,” 
Econometrica, Vol. 35, July-October, 1968, pp. 464-472, for more details. 
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entire log-normal distribution of costs. The endpoints highlighted above represent the range of possible 
values of cost in the sampling simulation.’ The frequency distribution illustrated on the Y-axis is the risk- 
adjusted cost estimate. 
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Figure 7-4: Risk-Adjusted Cost Estimate. 


7.6 COST AND BUDGET RISK 


7.6.1 Cost Risk 


The baseline cost estimate contains modelling uncertainty while the risk-adjusted cost estimate contains 
both modelling uncertainty and technical uncertainty and risk. The risk-adjusted probability distribution 
will therefore have a higher mean value and a higher variance than the baseline estimate. Its distribution 
will typically appear flatter and more skewed to the right, as Figure 7-5 shows. The difference in mean or 
expected values of the two distributions is cost risk. This value is usually expressed in monetary rather 
than percentage terms. It accounts for the cost impact of unfavourable outcomes in a major acquisition 
programme such growth in lines of code or failure of a new computer chip to work correctly. Aggregate 
cost risk can be allocated to any cost breakdown structure element, as appropriate. 


° More precisely, since the log-normal distribution extends to infinity, upper-bound costs illustrated here should be regarded as 
holding for some fixed percentile, such as the 99". 
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Figure 7-5: Cost Risk Analysis Output. 


7.6.2 Budget Risk 


Budget risk is the probability that the actual cost of a weapon system acquisition programme will end up 
exceeding a given budget, as Figure 7-6 shows. In this case, the budget is set at the mean value of the risk- 
adjusted cost estimate. Costs to the right of the mean are all legitimate possibilities, as are those to the left 
of the mean. Since the budget is finite, there’s a certain probability it will be exceeded. The percentage of 
the area of the distribution to the right of the budget is defined as budget risk. It is usually expressed as a 
number such a 40%, or 50%, or 60%. The value of risk analysis is that it quantifies this probability. 
Decision makers can then determine what degree of risk to accept, given the value of the weapon system 
and given the values and risks of alternative systems in a warfighting portfolio. A low budget implies a 
high probability of an overrun while a high budget implies a low probability of an overrun. It’s up to the 
decision maker to decide where he/she wants to set the budget. 
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Figure 7-6: Budget Risk Analysis Output. 
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7.7 SENSITIVITY ANALYSIS 


Another popular method for taking risk into account in generating life-cycle cost estimates is sensitivity 
analysis. This process, in its simplest form, measures the impact on cost of changing one or more key 
input values about which there is uncertainty. For example, a pessimistic, baseline, and optimistic value 
might be chosen for an uncertain variable, such as the slope of a learning curve. Then, an analysis could be 
performed to see how the life-cycle cost of the weapon system changes as each of the three chosen values 
is considered in turn, with all other factors held the same. 


Further, a more complex analysis could be undertaken where values associated with several variables such 
as learning curve rate, degree of hardware commonality, and growth of out-year business base are grouped 
or bundled together to form pessimistic, baseline, and optimistic scenarios. For each scenario, a cost 
estimate is then generated. 


A similar analysis can be conducted by varying a set of parameters from low to high and determining the 
impact on total cost of each. Results of this kind of analysis are often displayed in a tornado chart, 
as shown in Figure 7-7, where parameters are ranked according to their relative influence on cost. 
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Figure 7-7: Example of Sensitivity Analysis Tornado Chart. 


Finally, a shortcoming of sensitivity analysis is that it makes no statement about the probability that a low, 
baseline, or high value of a parameter might be obtained. Nevertheless, the technique can be performed 
rather quickly and inexpensively. Experience suggests that in many concrete applications it provides a 
reasonable and relatively trustworthy estimate of life-cycle costs. 


7.8 FINDINGS 


This sub-section describes a summary of the methods and models related to risk and uncertainty that are 
being used by the participating nations, based on the analysis of the matrices completed by the study 
participants and introduced in Chapter 1. Figure 7-8 shows the results of this analysis graphically. 
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Figure 7-8: Summary of Methods and Models Used to Measure Risk and Uncertainty. 


The findings at Figure 7-8 appear to show that risk and uncertainty analysis is widely used by NATO and 
PfP-nations. We all say that we are doing this. However, this figure does not present the whole story. 
Discussions in our Task Group showed something completely different. All nations are familiar with 
methods and models to be used for quantifying risk and uncertainty, but among NATO and PfP nations, 
the application of risk and uncertainty analysis varies widely. No NATO standard procedure is used or 
recommended. Collectively, the group of nations is certainly not guilty of Emerson’s aphorism that 
“a foolish consistency is the hobgoblin of little minds'””’. Just the opposite seems to hold. We are widely 
inconsistent in the application of risk and uncertainty methods between nations, within nations, and even 
sometimes within the same organisation in a single nation! 


More specifically, the survey results indicate that, sometimes, risk and uncertainty analysis is not 
undertaken at all in generating a life cycle cost estimate. Instead, single, point estimates are provided to 
the decision makers. At other times, when risk and uncertainty analysis is conducted, the two most 
commonly used techniques seem to be expert opinion and sensitivity analysis. Detailed risk and 
uncertainty modelling, such as Monte Carlo simulation, seems to be undertaken, with varying degree of 
frequency, by only two to three nations. 


7.9 RECOMMENDED PRESENTATION OF RESULTS 


Figure 7-9 presents a recommended approach for communicating results of a life cycle cost estimate to 
senior decision makers.'' The top line shows a three point range of estimates, and conveys the idea that a 
cost estimate is not a single number, but rather a continuum or distribution of possible values. 


: Essays (First Series) “Self-Reliance” by Ralph Waldo Emerson (1803-1882): “A foolish consistency is the hobgoblin of 
little minds, adored by little statesmen and philosophers and divines.” 


"UK. Ministry of Defence and Impossible Certainty, RAND, 2006, pp. 84-86. 
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Figure 7-9: Recommended Presentation of Cost Estimating Risk Analysis. 


Analysts can use one or more estimation techniques in performing risk and uncertainty analysis. Some of 
these are shown in the top two bars or sections of the figure. The bottom section, which should always be 
included in the presentation of the estimate, shows all of the assumptions or scenarios associated with the 
low, baseline, and high estimates. Including this section enables decision makers to see clearly the cost 
implications of events that can influence the outcome of an acquisition programme. 


7.10 REFERENCES 


[88] 


[89] 


[90] 


[91] 


[92] 


[93] 
[94] 
[95] 


[96] 


See www.dau.mil/gdbks/risk_management.asp to obtain a copy of “Risk Management Guide for 
[U.S.] DoD Acquisition”. 


See http://www.ispa-cost.org/PEIWeb/newbook.htm to obtain a copy of the “Parametric Estimating 
Handbook” from the International Society of Parametric Analysts. See Chapter 11 for details on risk 


analysis. 


See http://www.scea-alabama.org/pubs.shtml to obtain a copy of “Cost Risk Analysis Without 
Statistics,” Garvey, Paul, MITRE, October 2003. 


See http://www.sceaonline.org for details on how to order a moderately priced copy of “Cost 
Programmed Review of Fundamentals” (CostProf) from the Society of Cost Estimating and Analysis. 
See Chapter 9 for details on cost risk analysis. 


See http://ceh.nasa.gov/ to obtain a copy of NASA’s “Cost Estimating Handbook.” Various sections 
relate to risk and uncertainty analysis. 


Risk Analysis — A Quantitative Guide 2nd Edition, Dr David Vose, John Wiley & Sons, 2002. 
Project Risk Analysis and Management 2nd Edition, Association of Project Management, 2006. 
Impossible Certainty, Cost Risk Analysis for Air Force Systems, RAND Corporation, 2006. 


Daneshkhah, A.R. (2004), Uncertainty in Probabilistic Risk Assessment, University of Sheffield. 


RTO-TR-SAS-054 7-13 


UNCERTAINTY ANDRISK een 
Appendix 1: Optimism Bias 


In the absence of the time or resources required to perform the detailed analysis prescribed in this chapter, 
an alternative is to apply percentages to a bottom-line estimate, as shown below. 


There is a demonstrated, systematic tendency for project managers to be overly optimistic. To redress this 
tendency cost analysts should make empirically-based adjustments to the estimates of a programme’s 
costs, benefits, and duration. 


These adjustments should be based on data from past, similar projects, and calibrated for the unique 
characteristics of the programme at hand. In the absence of a more specific evidence base, analysts are 
encouraged to collect data to inform future estimates of optimism, and in the meantime use the best 
available data. 


The main aim of applying this guidance is to provide a better estimate of the likely life cycle cost of a 
major weapon system acquisition programme. '” 


Table 7-1 shows the percentage build up when using an “optimism bias” approach. These percentages 


should be added to the cost estimate when the relevant programme information is immature or not 
available. 


Table 7-1: Optimism Bias Estimate Uplift Percentages 


Project Information Relevant 
Percentage Uplift 

Complexity of contract 7% 
Late contractor involvement in design 7% 
Poor contractor capabilities 4% 
Information Management 5% 
Design complexity 10% 
Degree of innovation 17% 
Inadequacy of business case 18% 
Poor management team 5% 
Poor project intelligence 4% 
Legislation/regulations 5% 
Technology 18% 


Figure 7-10 provides an illustration on the application of “optimism bias” in a programme review. 
It compares the known quality of the data Gudged from the list above) added to the programme estimates 
against the stochastic output from a cost risk analysis utilising a three point estimate. 


'2 A detailed description of the recommended adjustment ranges and a detailed approach is described in the “Supplementary 
HM Treasury Green Book” guidance on optimism bias, UK. 
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Figure 7-10: Application of Optimism Bias. 
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Chapter 8 - OTHER RELATED LIFE CYCLE COSTING ISSUES 


This chapter identifies and discusses other issues that the cost estimator/analyst will have to consider in 
conducting a life cycle cost estimate. 


8.1 TAXATION 


Income tax and other tax effects on capital investments are significant and should be considered carefully 
from the early phases of a system’s life cycle (e.g. during feasibility studies). Taxes are assessed as a 
percentage of taxable income, usually on a graduated basis: the higher the taxable income, the higher the 
tax rate. Taxes reduce the income from investments. 


A typical example for taxation, first introduced in France in 1954 and later extended to several Western 
European countries is the Value-Added Tax (VAT). The following is an example of the application: as a 
material moves from the raw material stage to the finished product, whenever it changes ownership, 
the product is taxed at a fixed rate on the value of the product after all the costs of manufacture (materials 
and other expenses) have been deducted. 


The advice from legal advisors at the NC3A (NATO Command, Control and Consultation Agency) 
suggest that the following statement is included in the Contract Conditions. 


e Bidders are informed that the Purchaser, by virtue of his status stipulated in the provisions of the 
NC30 (NATO Consultation, Command and Control Organisation) Charter, Article 67(c)(3), 
is exempt from all direct taxes (including VAT), and all customs duties on merchandise imported 
or exported. This provision reads as follows: 


“Each participating nation undertakes to grant to NC3A under the terms of Articles 9 and 10 
of the Ottawa Agreement, exemption from all direct taxes (except rates, taxes and dues which 
are no more than charges for public utility services) from the taxes on the sale of movable 
and immovable properties, and from customs and excise duties in respect of equipment 
imported or exported by NC3A or its appointed agents.” 


e The bidder shall therefore exclude from its price quotation all taxes, duties and customs charges 
from which the purchaser is exempted by international agreement. This practice could however 
differ among the member nations. 


The text above provides guidance on how NATO is treating the issue of taxation within NATO 
programmes. However, in individual nations, the regulations may be different and it is recommended 
that advice should be sought with the appropriate national authorities. 


8.2 DISCOUNTING AND PRESENT VALUE 


8.2.1 Discounting 


Discounting is a technique used to compare costs and benefits that occur in different time periods. It is a 
separate concept from inflation, and is based on the principle that, generally, people prefer to receive 
goods and services now rather than later. This is known as time preference. 


Most cost comparison techniques take into consideration the time value of money: e.g. a Euro today is 


worth some amount less in the future. For comparison purposes, future expenditure, occurring at different 
points in time, must be adjusted to a common point in time. This adjustment to a common point in time is 
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called discounting or present value analysis. Discount factors can be calculated once an interest rate and 
period of analysis is determined. These discount factors indicate the present value (today’s value) of a 
Euro, Dollar. GB Pound, etc., expended at the end of each respective year (assuming end of year 
discounting is used). Therefore, discounting converts various cash flows occurring over time to equivalent 
amounts for differing interest costs and the time value of money resulting from expenditures over varying 
time periods. 


Recommended discount rates are usually determined by a nation’s Treasury Department and must be used 
consistently on all programmes. Calculating the present value of the differences between the streams of 
costs and benefits provides the Net Present Value (NPV) of an option. The NPV is the primary criterion 
for deciding whether government action can be justified. 


8.2.2. Net Present Value 


The present value (PV) of a stream of expenditures is the sum of discounted Euros (or other currencies as 
applicable) over the life of a programme. The following data is needed before beginning the PV process: 


¢ Base Year. The base year is the year to which all costs are discounted. This is usually the same 
year that costs begin to accrue for any alternative. 


¢ Period of Analysis. The period of analysis is normally the time from the start of a programme to 
the end of the mission requirement. In most cases this will coincide with the economic life of a 
project as described below: 


e The economic life of a project is the period of time over which the benefits to be gained from 
a project may reasonably be expected to accrue. 


¢ Benefits from a project are limited ultimately by its physical life. This is the period a facility 
or piece of equipment can be used before it is exhausted in a physical sense, that is, unable to 
perform its stated mission. The economic life of a project is further limited by its 
technological life; that is, the period before improved technology makes the building, 
machine, etc., obsolete. Military or political considerations that may suggest benefit accrual 
for a much shorter period may further limit the economic life of a project. Ways in which 
economic life can be determined include policy, management judgement, Government or 
industry standards and experience. 


e In general, the economic life will be measured against a stipulated level of threat, or represent the 
period during which a given mission or function is required or can be supported. Once the base 
year and period of analysis has been determined, the PV is calculated using the following 
procedure: 


¢ Determine in what years the expenditures for the alternative will be made. 


e Select a discount rate appropriate to the period of analysis and list the discount factor for each 
year, using either year-end or mid-year discount factors. Specify and document which one is 
being used. 


¢ Multiply each yearly cost by its discount factor to get discounted currency for that year. 
Use the constant currency rate if your cost basis is in constant currency (as will normally be 
the case); otherwise use the current rate. 


e Sum the annual discounted currency to get a total PV of costs. Perform similar calculations 
for quantifiable benefits. The difference between the totals of PV benefits and costs will be 
the net present value of the project. 

e If there is a concern that the preferred alternative may change if a different discount rate is 
used, recalculate the results, varying the discount rate to see at what point (if ever) the 
preferred alternative changes. This will provide additional insight for the decision makers as 
they weigh the alternatives. 
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8.2.2.1 Example 


If the government discount rate is used to determine the level of government investment spending that will 
maximise inter-generational social welfare, the use of risk premiums (individuals, companies and market 
risk) are indisputable. However, it is a formidable task to implement such an approach because of 
problems associated with identifying the value of the social rate of time preference. Some alternative 
approaches are discussed below: 


e Step (a) — The use of the consumer rate of interest as the social rate of time preference can be 
justified only in very special circumstances; the ethical issues involved in choice of a social rate of 
time preference are not easily resolved. 


¢ Step (b) — There seems to be a consensus that there can be a multiplicity of social discount rates 
depending on the nature of the finances, risks, and the degree of spill over effects of a given 
programme. 


e The opportunity cost school approach can be derived as a special case of trying to introduce 
market risk principals. 


e The shadow price approach is formally equivalent to the opportunity cost approach. Although it 
uses a uniform discount rate (social rate of time preference), the approach adjusts the special 
features of individual cases by the choice of multipliers to compute consumption-equivalent costs 
and benefits. 


On the other hand, if the discount rate is used to filter government programmes rather than to determine 
the level of government investment expenditure needed to reach some societal optimum, then the existing 
theories do not adequately address the problem. In this case, the government opportunity cost rate 
(risk free rate/State bonds [say 10 years]) is the better choice. 


The role of government in (a) is highly idealised. In reality, the level of government investment spending 
is the result of a complicated political process in which economic efficiency is only one of many factors at 
work. The recommended approach is step (b). 


8.3 EQUIVALENT ANNUALISED COSTS 


Annual value is a measure of costs in terms of equivalent equal payments made on an annual basis. 
The equivalent annual cost of a programme is calculated by dividing the net present value of the 
programme by the cumulative discount factor for the number of years of programme life. An alternative 
approach would be to multiply the net present value by the appropriate annuity factor, where the annuity 
factor is the reciprocal of the cumulative discount factor. 


Annual value analysis often involves fewer calculations than present value analysis if differing lives are 
under consideration for investment alternatives, because annual value implicitly assumes equal 
replacement values and the least common multiple of the different lives without extra calculations. 
Consequently, the major advantage of the annual value method of comparing alternatives on the basis of 
periodic payments is that the complication of unequal lives of competing alternatives is automatically 
taken into account and the same selection decision as the present value technique will be yielded. 


For example, consider the appraisal of a programme which has examined two options to achieve the 
required objective: 


¢ The NPV cost of Option A over a5 year life is £10 m. 
¢ The NPV cost of Option B over a7 year life is £12 m. 
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The EAC (Equivalent Annual Cost) of Option A is calculated by dividing the NPV of £10 m by the 
cumulative discount factor for 5 years of 4.5151 (assuming a discount rate of 3.5%) to give £ 2.215 m. 
Alternatively, the NPV of Option A could have been multiplied by the 5 year annuity factor of 0.2215 
(assuming a discount rate of 3.5%) to give the same EAC of £ 2.215 m. 


The equivalent annual cost of Option B using the cumulative discount factor of 6.1145 or the 7 year 
annuity factor of 0.1635 is £ 1.962 m. 


Option A has the lower NPV cost. However, selecting the preferred option by the NPV criteria would 
result in a poor choice. As the two options have different lives, it is important to base the decision on 
Equivalent Annual Costs in order to compare like with like. Option B should be selected as it has the 
lower EAC. 


8.4 VARIATION OF PRICE 


Under conditions specified in contracts, there may be provision for revision of contract prices. A typical 
occurrence is a contract of a duration longer than a specified limit (e.g. 2 years), concerning the 
procurement of advanced technology defence systems through an international cooperation programme. 


In such cases, prices agreed at certain economic conditions are escalated to the period of actual 
performance. 


The formulae to be used for this purpose contain a material and wage/salary related portion of the price as 
well as indices in accordance with which the basic values have to be escalated. Price indices are 
percentages of annual increases in costs for a given economic sector. In the different countries, such price 
indices by sectors are published by governments or by specialised trade and industry associations, with an 
official acknowledgment. 


As soon as the minimum specified duration is exceeded, contract prices can be revised taking into account 
a price index applicable to labour rates (e.g. in connection to productivity improvements and in accordance 
with the annual increase in wages and salaries) and a price index applicable to specified classes of 
materials (e.g. steel, plastic, etc.). The source and the specificity of the indexes used must be clearly 
established at the time of the contract. 


The following equation provides a generic price revision formula: 


P; = (Po /100) * [a+b * (My / Mo) +c * (Ly /L 9)] 


where: 
Po = Initial price as stipulated in the contract 
P) = Price payable under revision clause 
a = Percentage of price excluded from adjustment (e.g. 5 to 15 per cent) 
b = Average percentage of labour cost (e.g. 45 to 65 per cent) 
Mo = Price index of specified materials (initial date) 
M) 7 Price index of specified materials (revised) 
c — Average percentage of materials cost (= 100 —a-— b) 
Lo = Average labour rate (initial date) 
lL] = Average labour rate (date of revision) 
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Mathematical formulae used in price revision can be perceived as a reasonable compromise between 
parties in a long-term agreement, as well as an interesting tool to control future costs. Fixed elements 


within the price revision formulae have to be negotiated on a case by case basis, e.g. taking into account 
inflation constraints. 
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Chapter 9 - NEW DEVELOPMENTS IN LCC ANALYSIS 


Over the duration of the SAS-054 study a number of associated requirements that will impact on cost 
analysis have been identified. However, although there is some understanding at this stage, the actual 
requirements and therefore the likely scope of the change in terms of the demand on cost analysis is not 
clearly known. 


The topic of joint war-fighting' (or operational activities) is becoming more important to NATO. 
At present, there is insufficient information on how to evaluate the situation where a number of discrete 
assets share the information/data to provide a total capability solution. The costing of the assets 
themselves is straightforward, but when combined the interpretation on apportionment where multi- 
mission systems feed into several capabilities is not clear. 


To more effectively manage scarce defence resources, several nations are initiating efforts to analyse the 
costs, capabilities, and risks of entire portfolios of assets in a joint war-fighting environment.’ 


Viewing capabilities across the entire portfolio of assets enables decision makers to make better informed 
choices about how to reallocate resources with the ultimate goal of delivering needed capabilities to the 
joint force more rapidly and efficiently*. Capability portfolios are intended to serve as a basis for strategic 
level trades by senior decision makers, as depicted here in Figure 9-1. 
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Figure 9-1: Prioritising using Portfolio Analysis. 


' NATO report entitled ‘Backgrounder’ interoperability for joint operations. 


* The 2001 Quadrennial Defense Review (QDR) in the United States directed DoD to shift from threat-based planning to 
capability-based planning. Further, the 2006 QDR reaffirmed the shift to capabilities-based planning and directed the use of 
joint capability portfolios to manage DoD resources. 

> Draft letters from the Undersecretary of Defense [U.S.] for Acquisition, Technology, and Logistics on four pilot programs for 
conducting portfolio capability analyses, July 2006. 
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In the United States, four pilot programmes or test cases are currently underway for performing portfolio 
capability analyses: Joint Logistics, Joint Battlespace Awareness, Joint Command and Control, and Joint 
Net-Centric Operations. Further, last year, the Department of the Navy in the United States conducted a 
pilot programme for portfolio analysis of mine countermeasure assets. 


Portfolio analysis, according to one of several possible implementations, would be conducted in 
conjunction with a concept decision review early in the planning, programming, and budgeting cycle, 
that is, prior to concept refinement in the U.S. acquisition framework. In this paradigm, an entire portfolio 
of assets, large and small alike, would be analysed. Gaps would be identified by the combatant commands 
(war-fighters) and trade-space would be offered as an option for at least partly funding these gaps. Further, 
ideally, a menu of portfolios would be generated from which senior decision makers could choose to meet 
strategic requirements. 
In terms of cost estimating for NATO and PfP countries, these guidelines are suggested: 

e Generate costs in constant-year monetary units (e.g. dollars or Euros). 

¢ Generate life cycle costs. 


¢ Depending on where a system or concept falls in the life cycle, this might include estimates of 
science and technology costs, development and production costs, operating and support costs, 
and disposal costs. 


¢ Generate costs for a ten or twenty year period. 


e Ensure that an entire portfolio of war-fighting assets is included in the analysis. Small systems are 
important, too, and completeness is essential. 


¢ Perform risk and uncertainty analysis. 
For the NATO and PfP cost community, this shift in emphasis from the analysis of individual programmes 


at key gates or milestones to the analysis of entire portfolios of assets, some of which are merely concepts 
rather than programmes, will entail: 


e Earlier involvement in the planning process. 


¢ Use of high level cost estimating relationships based on technical and performance characteristics 
of proposed systems. 


e Extensive data gathering as well as the creation of new databases linking systems to capabilities. 


¢ Use of new methods and models for analysing the costs, capabilities, and risks of a large group of 
proposed and existing assets rather than individual systems. 


It is recommended that further study be conducted to provide a better understanding of the processes 
and application to the benefit of NATO and PfP nations. 


9.1 REFERENCE 


[97] Transformation of analytical tools — using portfolio analysis techniques in defence applications, 
Capt J. Field USN, and Dr B. Flynn, DoD Comptroller magazine, September 2005. 
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Chapter 10 - ENHANCE WORK SAS-028: CBS 


10.1 INTRODUCTION 


The SAS-028 Task Group related to cost structures and life cycle costs for military systems developed a 
NATO generic cost breakdown structure and associated definitions that can be used by any military 
programme to construct its own bespoke and programme specific cost breakdown structure. 


It has been found that most nations have not adopted the generic cost breakdown structure reported in 
SAS-028 as their national life cycle cost breakdown structure. However, the NATO generic cost 
breakdown structure has been applied on specific programmes and some areas of enhancement are 
suggested. 


10.2 RATIONALE 


The application of the outputs of the report has shown that some difficulties had been met when the 
aggregation of the cost elements was performed (application of the construction of CBS Chapter 12 of the 
RTO-TR-058 report). In fact, the coding proposed in the report during the identification process of the 
cost elements (Chapter 9 of the RTO-TR-058 report) allowed the definition of a CDB (cost database). 
This coding was very useful for the identification of the three dimensions (Activity, Product and 
Resource) and the phase of the programme, but it could not easily define particular positions of specific 
cost elements inside the cost breakdown structure. 


Although the NATO GCBS coding was a useful starting point it appeared that the different stakeholders 
evaluating the life cycle costs also needed (due to financial provisioning) a simple codification in order to 
identify where the various cost elements were within the cost breakdown structure. The following example 
provides an explanation on this issue: 


Consider the cost element ‘total contractor labour costs for manufacturing rework on the air vehicle’s 
air/speed brakes’, it is coded 6.1.1.5.2.1.1.1.1.8 (see Chapter 9 of SAS-028 report): 

e The first field is related to the Phase (Production generic code 6). 

e The second field is related to the Resource (Contractor Labour generic code ‘1.1’). 

e The third field is related to the Activity (Rework Modification generic code ‘5.2’). 

e The fourth field is related to the Product (Main System generic code ‘1’). 

¢ The fifth field is related to the Product Detail (Air Speed Brake standard or customised code, here, 

1.1.8). 

This coding structure does not help the non-expert to understand or find where a specific cost element is 
within the overall cost breakdown structure. It is therefore proposed to create: 

e A ‘generic hierarchy’ of the cost breakdown structure with an incremental coding. 


e A new presentation of the generic cost breakdown structure form allowing the link between the 
CDB and cost breakdown structure coding. 


10.3 GENERIC HIERARCHY 


This generic hierarchy is based on the cost breakdown structure activities defined in Chapter 8 of the 
RTO-TR-058 report. Figure 10-1 shows this generic hierarchy. The cost aggregates are defined up to the 
second level. 
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Figure 10-1: Proposed Generic Hierarchy Cost Breakdown Structure. 
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Once this generic hierarchy and its related coding have been set up, the different cost elements can be 
assigned to the appropriate cost aggregate. A coding related to the cost breakdown structure (incremental) 
could therefore be attributed to these cost elements. 


Consider the same example as above and assume that this cost element is the first of the appropriate cost 
aggregate, it could be coded 6.7.1 in the cost breakdown structure as it is related to the production 
phase/system test and evaluation (6.7). Therefore, for communication purposes between the stakeholders, 
the codification would be 6.7.1 instead of 6.1.1.5.2.1.1.1.1.8. 


If this structure is not uniformly adopted by all the nations then some form of mapping will have to be 
conducted to meet all the national and multi-national life cycle cost reporting requirements. 


10.4 GENERIC COST BREAKDOWN STRUCTURE FORM 


In the SAS-028 report, there are two possible types of presentation (table and list) proposed. It is 
suggested that the introduction of the above described coding in the table form is conducted as well as the 
identification of the different dimensions for each cost element. The figure below shows an example of 
this presentation. This presentation is given for a specific cost aggregate, but it is also applicable to the 
other cost aggregates. 
Consider the in-service phase/maintenance (7.3) and then consider three different cost elements: 

e Level 1 maintenance government labour cost for the main system. 


¢ Level 3 maintenance contractor labour cost for the main system. 


e Level 3 maintenance contractor material cost for the main system. 


The presentation for this aggregate is given in the following matrix: 


Phase | Resource | Activity | Product 


7. In Service Phase 


7.3 Maintenance 


evel 1 government Labour 


CBS coding 


CDB coding 


evel 3 Contractor Labour 


evel 3 Contractor Material 


Figure 10-2: Example of Aggregated Cost Elements. 


10.5 DIMENSIONS 


Leading on from the presentation above, it is suggested that an additional dimension could take into 
account the nation dimension in multi-national programmes. The cost sharing between the nations should 
be identified in the life cycle cost model, therefore the distinction of the cost elements according to the 
nations should be set up in the cost breakdown structure. 
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In addition, in order to be more precise in terms of time, it is suggested that customisation for each phase 
dimension in the payment profiles should be determined in accordance with the specific requirements of 
the programme. 


For example, during the production phase, the deliveries of the systems could last many years (particularly 
in multi-national programmes), therefore the payment schedule could be different from one delivery to 
another. This would lead to different payment profiles related to the different deliveries. 


Taking account of these profiles within the cost breakdown structure could improve the presentation of 
individual budget forecasts. 


10.6 RECOMMENDED ENHANCEMENTS TO THE NATO GCBS 


The NATO Generic Cost Breakdown Structure developed by the SAS-028 Task Group does not allow the 
identification of the Life Cycle Cost results over time and the National contribution in case of multi- 
national programmes. Therefore, it is recommended to include two dimensions in addition to the Activity, 
Product and Resource dimensions: 


e Time phasing; and 


e National contribution. 


As the coding of the Generic Cost Breakdown is complex for non-experts, it is recommended to adopt a 
Generic Hierarchy for the GCBS. 
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Chapter 11 - CONCLUSIONS 


11.1 GENERAL 


The results of life cycle costing must, whatever the phase of the programme, contribute to the process by 
which managers can make the best decisions on options presented to them. These options can include 
evaluation of future expenditure, comparison between alternative solutions, management of existing 
budgets, options for procurement and evaluation of cost reduction opportunities. Life cycle costing is also 
used for affordability assessment and determining the cost drivers associated with the Key Performance 
Indicators or Key User Requirements. 


There are many methods and models available to conduct life cycle cost analysis. It was important to 
understand the applicability and boundaries of each method and model in order to use them appropriately. 
The core objective of the SAS-054 Task Group was to understand NATO and PfP nations’ methods and 
models and to promulgate best practice throughout the life cycle. 


11.2 REVIEW OF COST FORECASTING METHODS 


Most cost estimates require the use of a variety of methods. A different approach may be used for each 
area of the estimate so that the total system methodology represents a combination of methods. Sometimes 
a second method may be used to validate the estimate. 


When choosing an estimating method, the cost estimator must always remember that cost estimating is a 
forecast of future costs based on a logical interpretation of available data. Therefore, availability of data 
will be a major factor in the estimator’s choice of estimating methodology. 


The best combination of estimating methods is one which makes the best possible use of the most recent 
and applicable historical data and systems description information and which follows sound logic to 
extrapolate from historical cost data to estimated costs for future activities. 


The report has captured all the key estimating methods and provided examples to demonstrate their 
applicability. For consistency, both the methods and models have been categorised as Optimisation, 
Simulation, Estimation and Decision Support. 


The findings at Figure 4-4 clearly show that in order to generate a cost estimate all participating nations 
use many methods across each of the phases considered. 


The findings confirm that all nations used a similar process to develop life cycle cost estimates; that the 
quality of the available data nearly always determines the method to be employed. In addition the type of 
study (strategic planning, options analysis, simulation and traditional estimating) and the life cycle phase 
also influence the process and the appropriate estimating method. 


11.339 REVIEW OF COST FORECASTING MODELS 


All nations use some form of in-house developed model for life cycle cost analysis. The majority also use 
commercial cost estimating models. A few nations do not use any type of commercial model; instead they 
rely totally on in-house developed models and/or other types of methods such as expert analysis. 


Nearly 40 different models have been identified in the matrices. Almost half of them are commercial 
models and the remainder have been developed in-house. There are differences between the in-house 
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models, but one common feature is that many are developed in a spreadsheet environment and are often 
tailored for each specific programme. The in-house models are generally only used by the nation that has 
developed the model. 


The findings at Figure 5-1 show clearly that models for estimation are the preferred models for life cycle 
cost analysis throughout the phases. In addition to models for estimation, models for decision support are 
used in the earliest phases and models for optimisation and simulation later on. 


In order to provide verification of the life cycle cost estimate, it is good practice to use more than one 
model. Where sufficient data is available, the use of models for simulation and/or optimisation to 
supplement the overall life cycle cost model should be adopted. The use of multiple methods and models 
should always be evaluated on a cost-benefit basis ensuring that the added value provided from life cycle 
cost analysis is maintained. 


11.4 GUIDELINES FOR THE COLLECTION AND UNDERSTANDING OF 
COST RELATED DATA FOR NATIONAL AND MULTI-NATIONAL 
PROGRAMMES 


In terms of time, effort, and resources consumed, collection of data is a major part of a life cycle cost 
study. Life cycle costing is a data driven process, as the amount, quality and other characteristics of the 
available data often define what methods and models can be applied, what analyses can be performed, 
and the results that can be achieved. 


As a system progresses through the life cycle, the types of data available evolve in a number of ways. 
As this in turn defines the task of data collection and the life cycle costing process in general, it is 
important to be conscious of these developments. 


First and foremost, the amount of data available increases as the system becomes better defined. 
Obviously, very little is known about the end system when a project begins and all that exists is an 
identified capability gap or a general concept, whereas, when a system is in-service, the system and its 
environment can be documented in almost infinite detail. 


Unfortunately, because uncertainty, risks, and opportunities decrease as the life cycle progresses, the need 
for knowledge is greatest at the earliest stages. This means that more time and resources should be 
allocated to the data collection effort during the earlier stages of the life cycle in order to develop an 
acceptable and auditable life cycle cost estimate. 


Exchange of data between ERP systems and databases can be a cumbersome and time consuming affair if 
data formats and data models differ. It is therefore beneficial to have generally accepted and well 
documented standards. One such is ISO 10303-239 which has been put forward by NATO to be adopted 
as a STANAG. In the long term, PLCS has the potential to be an important tool to help collect and 
exchange high quality, well documented data. However, the PLCS is a very large and technically complex 
mechanism, and implementing PLCS would be a huge undertaking for any organisation. Alternative, 
ad hoc solutions in the form of agreed upon and documented templates, etc., may be used, but this makes 
the data harder to use at a later date for other projects or purposes. When possible, officially defined and 
accepted standards are preferred. 


It is recommended to have previously agreed upon and well documented templates or standards for data to 
be provided by contractors and suppliers. Furthermore, particular care must be taken to secure that the data 
received from contractors, or other sources with a vested interest in a programme, are accurate and 
unbiased. The UK MDAL (Master Data and Assumptions List) is one well documented mechanism for 
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ensuring that all stakeholders buy into a common and clearly stated understanding of the project and 
the system of interest. A similar function is fulfilled for major acquisitions of the US DoD by the CARD 
(Cost Analysis Requirements Description). Furthermore, the US DoD has developed the CCDR 
(Contractor Cost Data Reporting) and SRDR (Software Resources Data Report) systems for accumulating 
actual contractor costs necessary to analyse costs efficiently and effectively. While data collection through 
formal reports such as CCDRs and SRDRs is extremely important and beneficial, there is still no 
substitute for taking the time to understand and verify the accuracy of historical information, and the 
programmatic context in which it was obtained. 


11.5 TREATMENT OF UNCERTAINTY AND RISK 


History has shown that cost estimates tend to be low — only about 10% of all programmes come in under 
or on budget. Because of this, cost estimates need to be adjusted for risk. Mistakes happen during 
estimation, schedules slip, technical difficulties arise, assumptions prove false, missions change, or the 
proposed hardware or software solution turns out not to meet the needs of the joint war-fighter. Because of 
all of these factors, risk needs to be included so that the analyst truly conveys to decision makers the 
uncertain nature of the estimate. 


There are a wide variety of methods and models available for conducting risk and uncertainty analysis of 
life cycle cost estimates of weapons systems. These include sensitivity analysis, risk registers from the 
U.K., and detailed Monte Carlo simulation. Each, if used properly, can yield scientifically sound results. 
However, based on our collective experience in cost estimating within the government and in the private 
sector, it’s fair to say that the sophistication and underlying theory of many popular models often far 
exceeds the quality of the basic data inputs. There is simply no substitute for taking the time and effort to 
understand the technical risks and challenges in developing and producing sophisticated defence systems. 


The most important part of the process of estimating risk and uncertainty, and probably the most difficult, 
is data collection and analysis. All the variables in the cost estimating model potentially affected by risk 
and uncertainty first need to be identified. These variables often include simple ratios and factors as well 
as more sophisticated cost estimating relationships (CERs) based on regression analysis. For Monte Carlo 
simulation, probability distributions need to be estimated or selected for each variable. 


In terms of presenting results to senior decision makers, we highly recommend the use of a standard 
format which includes use of a three-point scale to convey the idea that a cost estimate is not a single 
number but rather a continuum or distribution of values. Assumptions or scenarios associated with low, 
baseline, and high estimates should be stated to enable decision makers to see clearly the cost implications 
of events that can influence the outcome of an acquisition programme. 


11.6 GUIDELINE FOR MULTI-NATIONAL PROGRAMMES 


The life cycle cost studies for multi-national programmes follow the same principles as defined for 
national programmes. Nevertheless, there are some specifics that should be taken into account in terms of 
organisation, types of life cycle cost studies, cost models and presentation of results. 


In terms of organisation, life cycle cost studies should be co-ordinated centrally by the pilot nation or the 
IPO or the NATO agency following the multi-national structure adopted. One or more participant(s) could 
perform a peer review including a verification and validation of the life cycle cost studies performed 
above. 


The type of life cycle cost studies could be focused on the assessment of different alternatives related to 
commonality part. For this purpose, a specific process is necessary to identify the areas in which 
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alternatives could be defined and assessed. Figure 2-16 shows a process for the selection of the areas and 
the definition of the related scenario. 


The basic principle for these studies is that they should be based on an agreed common life cycle cost 
framework. This implies the selection of a common model(s), the building of a common cost breakdown 
structure, the aggregation of the different outputs from the models, the definition of harmonised 
assumptions (in particular for deployment(s)) and a common process for data collection. 


The life cycle cost results should be presented in one currency. The choice of the currency should be made 
before the launch of the life cycle cost studies and the evolution of the exchange rates taken into account 
in the model. 


11.7 AWARENESS OF NEW DEVELOPMENTS IN METHODS AND MODELS 


This study identified that the topic of joint war-fighting is becoming more important to NATO and this 
will influence the life cycle costing requirements. At present, there is insufficient information on how to 
evaluate the situation where a number of discrete assets share the information/data to provide a total 
capability solution. The costing of the assets themselves is straightforward, but when combined the 
interpretation on apportionment where multi-mission systems feed into several capabilities is not clear. 


Viewing capabilities across the entire portfolio of assets enables decision makers to make better informed 
choices about how to reallocate resources with the ultimate goal of delivering needed capabilities to the 
joint force more rapidly and efficiently. Capability portfolios are intended to serve as a basis for strategic 
level trades by senior decision makers and it is essential the life cycle costing plays a major role in 
supporting this new development. 


For the NATO and PfP cost community, this will cause a shift in emphasis from the analysis of individual 
programmes at key gates or milestones to the analysis of entire portfolios of assets. This will require 
further investigation and examination to assess the impact. 


11.8 REVIEW AND POSSIBLE ENHANCEMENT OF NATO GENERIC COST 
BREAKDOWN STRUCTURE 


The SAS-028 Task Group related to cost structures and life cycle costs for military systems developed a 
NATO generic cost breakdown structure and associated definitions that can be used by any military 
programme to construct its own cost breakdown structure. The outputs of this Task Group have been 
applied on specific programmes and some areas of enhancement have been suggested. 
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Chapter 12 -RECOMMENDATIONS 


12.1 KEY RECOMMENDATIONS 


The following key recommendations are made with regard to the development and improvement in life 
cycle costing for multi-national programmes. 


12.1.1 Life Cycle Costing Methods 


1) Life cycle cost estimates should be fully documented (Sub-section 2.3.2) 


A cost analyst should be able to re-create the complete estimate working from the documentation 
alone. 


All assumptions and data related to the study should be captured in an MDAL or CARD or similar 
document. 


Assumptions recorded in an assumptions list such as the MDAL or CARD should be questioned 
by an independent technical team. 


2) All life cycle cost estimates should be prepared by suitably experienced personnel (Sub-section 
5.2.3.1) 


Decisions such as budget setting and options analysis studies are often conducted when data to 
support cost forecasting and life cycle costing is very sparse. It is therefore essential that 
experienced personnel are used to conduct the life cycle cost estimates to support the decision 
process at these key stages. 


3) The life cycle cost analysis should include an affordability analysis (Sub-section 2.9) 


Affordability plays an important part in programme decisions throughout the life cycle. Even before 
a programme is formally approved for initiation, affordability plays a key role in the identification 
of capability needs. This aspect is part of the process which balances cost versus performance and 
in establishing key performance parameters. Although this is not common practice in all nations the 
assessment of affordability is one that we recommend should be conducted by all nations. 


4) Life cycle cost estimates, where possible, should use two independent methods for each cost 
breakdown structure element (Sub-section 4.4) 


The use of two independent methods to develop the life cycle cost estimates will improve the 
confidence in the results and help to validate the outputs. It is accepted that this may be tempered 
by the constraints imposed by a financial threshold (see Sub-section 2.6) or by a simple 
consideration of what the estimate will be used for (e.g., rough cost for initial views or detailed 
costs for decision making). 


12.1.2 Life Cycle Cost Models 


5) All life cycle cost models should be validated (Sub-section 5.5) 


It is essential that all life cycle cost models implemented through spreadsheets or more advanced 
programming techniques be validated by using recognised testing processes. This will increase 
confidence that the model is fit for purpose and that the input data and results can be assessed 
through a clear audit trail and mathematical reasoning of any cost estimating relationships. 
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12.1.3 Data for Life Cycle Costing 


6) Investments should be made to increase the accuracy, visibility, and availability of cost, 
programmatic, technical, and performance data within the NATO/PfP cost analysis community 
(Chapter 6). 


¢ Data collection forms a large part of the life cycle costing activity and significant effort is 
expended to gather and analyse the data so that it is suitable for use in life cycle cost analysis 
studies. Improvements in data exchange standards or even the development of a NATO costing 
database would: 


° Improve the quality of the life cycle cost estimate; 
° Reduce the effort needed to conduct the life cycle cost estimate; and 
° Reduce the time schedule to conduct the life cycle cost estimate. 


12.1.4 Multi-National Programmes 


7) For multi-national programmes the participating nations should agree on a common LCC 
framework (Sub-section 2.10.6) 


¢ The life cycle cost studies for multi-national programmes follow the same principles as those 
required by a national study. However, there are some specifics that must be taken into account in 
terms of organisation, models and the presentation of results. It is essential that all parties in a 
multi-national programme agree on a common life cycle cost framework. This framework is 
determined by the costing boundary and the tools that will be employed to populate the 
framework. A common framework will provide consistency, comprehensiveness, traceability and 
audit. All are essential to achieve life cycle cost estimates in a timely and responsive manner. 


12.1.5 NATO Generic Cost Breakdown Structure 


8) Enhancements to the GCBS (generic cost breakdown structure) to improve its use (Chapter 10) 


¢ It has been found that most nations have not adopted the generic cost breakdown structure reported 
in RTO-TR-058 as their national life cycle cost breakdown structure. However, the NATO generic 
cost breakdown structure has been applied on specific multi-national programmes and some areas 
of enhancement are recommended. 


The current structure does not allow the identification of the life cycle cost results over the time 
phasing for national financial and programme contributions. Therefore, it is recommended to 
include two dimensions in addition to the Activity, Product and Resource dimensions. These 
additional dimensions are: 


e Time phasing; and 
¢ National contribution. 


As the coding of the Generic Cost Breakdown is complex for non-experts, it is recommended to 
adopt a Generic Hierarchy for the GCBS. 


12.1.6 Uncertainty and Risk 


9) Risk and uncertainty analysis should be conducted at the same time as the life cycle cost 
estimate (Sub-section 7.9) 


¢ Life cycle cost estimates of weapon system acquisition programmes are inherently uncertain and 
risky. To better support senior leadership, some sense of risk and uncertainty needs to be presented 
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at the same time as developing the point estimate. This will present the decision maker with a 
comprehensive true view of the programme’s likely eventual outcome. 


10) The results of a life cycle cost estimate should be shown as a three point range of estimates 
(Sub-section 7.9) 


A life cycle cost estimate is not a single number but rather a continuum or distribution of possible 
values. 


12.2) FURTHER RECOMMENDATIONS 


In addition to the key recommendations listed above it is further suggested that the other recommendations 
listed below are considered as each will help in the development and improvement in process and 
application of life cycle costing. 


We recommend that each nation sets its own financial threshold value for conducting life cycle 
cost studies and that this threshold should be determined in terms of total programme cost, 
political requirements and timeliness (Sub-section 2.6). 


To fully support the tender evaluation process, it is recommended that a life cycle cost 
questionnaire is issued with the tender documents so that the procurement agency can conduct an 
independent comparative life cycle cost evaluation on all the tenders. This will improve the 
understanding of the tender offer and provide a degree of credibility in the predicted life cycle 
costs results (Sub-section 2.8.3). 


We recommend that when supporting contractor submissions then all cost data and substantiating 
information is provided in a format that is clear, complete and ready for evaluation (Sub-section 
2.8.3). 


There is a clear need for all participating nations in a multi-national acquisition to understand and 
trust the cost models used. The participating nations will need to agree on a common life cycle 
costing framework (as detailed in the key recommendations). It is further recommended that clear 
guidelines are produced with regard to which data to use and how to collect this data (Sub-section 
2.10). 


The issue of a reference currency and currency exchange should be resolved prior to commencing 
any life cycle cost study. It is recommended that advice be sought from the recognised national 
economic advisor to ensure consistency and correctness in the application of the life cycle cost 
modelling (Sub-section 2.10.7). 


It is also recommended that each nation within a multi-national programme apply their own cost 
model and applicable data (CERs, labour rates, etc.) to arrive at its national cost estimate and that 
this information is used in the collective multi-national life cycle cost framework (Sub-section 
2.10.7). 


Prior to the start of production it is recommended that a joint (contractor and government) 
risk register is developed to support the assessment of the financial risk liability and to assist in 
the risk management and mitigation activities (Sub-section 3.7.6.1). 


During and Post Manufacture, it is recommended that all actual costs incurred by the contractor 
are certified. This data can subsequently be used to refine and calibrate future cost forecasting 
models (Sub-section 3.7.6.3). 


It is recommended that research is conducted continuously to enhance methods and models for 
life cycle costing (Sub-section 4.4). 


It is recommended that anticipation is made of future data requirements to support life cycle 
costing and that the data is collected accordingly (Sub-section 6.1). To achieve this, an agreement 
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on well-documented templates or standards for the data to be exchanged should be reached 
between all the stakeholders (Sub-section 6.4). 


In individual nations, the regulations relating to the interpretation and calculation of tax may be 
different. Should this be a requirement for life cycle costing then it is recommended that advice 
should be sought with the appropriate national authorities (Sub-section 8.1). 


Larger (more expensive) programmes will usually demand more effort and rigor. It is recommended 
that sufficient time and effort should be allowed in order to provide a robust cost estimate to meet 
the programme requirements. However, there should be a balance between estimating effort and the 
value of the programme (Sub-section 2.6). 


A uniform communications format should be used for presenting the life cycle cost estimates risk 
analysis. This will aid the decision makers in their evaluation of programmes by presenting the 
basis of the spread of the costs, the method employed to conduct the simulation and the provision 
of the underlying assumptions (Sub-section 7.8). 


FOLLOW-ON ACTIVITIES 


The following paragraphs outline recommendations for further studies that would benefit the understanding 
and use of life cycle costing in NATO and multi-national environments. 


12-4 


The next logical step on completion of SAS-054 would be to demonstrate the proof of concept 
(methods and models) described in the report by using a practical application of the guideline: 


e A typical example could be an existing NATO programme (but only using data that was 
available at the time) and/or any other multi-national programme (e.g. AWACS, AGS, JSF, 
NH-90, FREMM). 


Further research should be conducted in the area of capability portfolio analysis (see Chapter 9). 
This topic of joint warfare is becoming more important to NATO and, at present, there is 
insufficient information on how to evaluate the situation where a number of discrete assets share 
the information/data to provide a total capability solution. 


e An investigation into new methods and databases would support this requirement. 
Research into the life cycle costs of software. This report has not addressed software cost estimating 
as it was felt that this was a subject in its own right. Many academic studies are being conducted 


into open system architecture, modular construction and system behaviours that employ software 
intensive configurations. 


¢ Much is known about modern techniques in software development but the issue of assessing 
software quality, reliability and support costs is still vague. 

Life cycle cost estimates are produced for a variety of reasons. It would benefit the NATO 

community to investigate how the cost estimates are being used in the decision making process. 

¢ This could avoid the situation where enormous effort may be spent in generating cost 
estimates when the answer could have been given in a more simplistic and effective manner. 

Estimating accuracy has been an issue for many years. An evaluation could be conducted that 

studied the delta between the original cost estimates and the actual costs. 


¢ This would provide a benefit by having a definitive document that could provide a view of 
estimating accuracy across a number of procurement processes. 


RTO-TR-SAS-054 


OTAN 


RECOMMENDATIONS 


Research should be conducted continuously to enhance methods and models for life cycle costing. 


¢ Periodically, the US Department of Defense undertake an initiative to review the basis and 
techniques employed in cost estimating. This is supported by industry, a number of academic 
groups and learned societies. However, these initiatives purely examine techniques that will 
be employed within the US. It would be beneficial to conduct a similar continual review 
across NATO and PfP nations. 


The SAS-054 study gathered information on each nation’s approach and use of models to generate 
life cycle costs. The study did not get a clear comprehension on the range of the functionality that 
could be provided by some of these models. 


¢ It would be of benefit to look in more detail on how these life cycle costing models generate 
cost for Research and Development, Production, Operating and Support. 


The issue of calibration, verification and validation of cost estimating models is of paramount 
importance. However, little or limited space is given in handbooks on the requirements and 
methods of validating cost models. 


e A study could be initiated to develop a common methodology for validating cost models, this 
would help to ensure cost estimating consistency across NATO and PfP nations on each 
nation’s approach and use of models to generate life cycle costs. 


All life cycle cost estimates are only as good as the data that underpins the estimate. Much 
investment has been made in adopting ERP-systems to support financial and project reporting. 
The use of these systems in providing good quality data to support life cycle cost estimating is not 
clearly known. 


e A study should be conducted to evaluate the benefits or otherwise in adopting an ERP-system 
versus the investment in a bespoke system (e.g. VAMOSC) to assist the life cycle cost data 
collection process and improve cost estimating methods and accuracy. 
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Annex A —- NATION’S COMPLETED MATRIX 


INATO LCC Phases 


National LCC Phases 
Nation: Denmark 


Type of costing studies required 


Description of the process or national 
guideline to be followed - 
how are we going to do this ? 


What methods are used ? 


What models are available ? 
Commercial models 


What models are available ? 


In house developed models 


equirements to apply national guideline 


Requirements to apply methods 


Requirements to apply models 


Restrictions on applicable methods or models, 
depending on the goal 


How can data be collected 
How is risk and uncertainty considered ? 


What models and tools are available 
to assess uncertainty ? 


What models and tools are available 
for risk analysis? 


Requirements to apply risk methodology 


Abbreviations 


CBA = Cost Benefit Analysis 
CBS = Cost Breakdown Structure 
COO = Cost of Ownership 

EVM = Eamed Value Management 


1A = Investment Appraisal 
RFI = Request for Information 
RFQ = Request for Quotation 
SME = Subject Matter Expert 
UPG= Unit Production Cost 


[PHASES in LIFE CYCLE 


Mission need 
jevaluation phase 


| phase 


‘Acquisition phase 


Project definition phase 


Requirements statement (Basic Military Requirements, Operational User Requirements) 


Design and 


{Production 
[Development phase | 


phase 


exploration, Invite tender, Tender 
Type selection, Order) 


LCC-analysis to compare alternatives and establish budgets 


7 


In-service 
phase 


Phase in + Service and 
maintenance phase 


Follow-up on LCC-analysis to 
validate budgets and validate 
and improve estimating 
techniques 


Disposal phase 


Phase out 


Guideline/standard process is being developed 


Parametric 


Standard generic Excel based model (Under development), 
Various purpose-made Excel models 


model 


none none none none MOD requirement to develop and use guideline/standard process none 
none none none none Tone none none 
none Tone Tone Tone MOD requirement to develop and use generic framework Tone none 


Lack of expertise and experience. Low priority within MOD 


and Defence organisations. 


DeMars SAP R/3 ERP system (Service data for similar 
systems) 


DeMars SAP R/S ERP 
system (Service data) 


Expert opinion (very limited 


Expert opinion (very limited 


Expert opinion (very limited 


Expert opinion (very limited 


Expert opinion (very limited use), Sensitivity analysis 


Expert opinion (very limited 


Expert opinion (very limited 


use) use) use) use) use) use) 
Simple Monte Carlo functionality implemented in generic 
model 
none none none none Tone Tone none none 
No No No No No No No No 


[Commercial Models 


"In-House" model 
Model 


Methodology 


Generic Excel framework model (under development) 


Figure A-1: Completed Matrix from Denmark. 


High level CBS, Parametric 
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ANNEX A — NATION’S COMPLETED MATRIX 


Type of costing studies required 


Description of the process to be followed - 


What tools are available ? 
IN House Tools 


[PHASES in LIFE CYCLE 
SS) | SS Se ee SS ae en SSS 


Design ana 
Pre-feasibility Feasibility Project definition [Development 
phase phase 


phase phase 


Mission need 
levaluation phase 


In-service 
jphase 


Production 
phase 


préparation conception Réalisation Utilisation 


Disposal phase 


Démantélement 


choice of operational solution choice of technical solution 


Budget previsions 


1st LCC estimation by end of phase |enhanced LCC estimation in end of 
phase file 
FACET PRICE, COST+ 


MOPSOS (armored vehicule); SCOPE (ships) 
Excel based applications 


Budget planning 


Analogy, CERs Engineering 


mandatory at end of phase mandatory, whole along the phase desirable desirable 


SIPROG : compiles all DGA contracts 


Figure A-2: Completed Matrix from France. 
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ANNEX A — NATION’S COMPLETED MATRIX 


ISSUE 01 
Date 22 12 2004 


INATO LCC Phases 


National LCC Phases 
Nation Germany 


alternative * 


‘Type of costing studies required 
Description of the process to be followed - 
how are we going to do this ? 


What tools are available ? 
(Commercial Tools 


IN House Tools 
depending on the goal 


What models and tools are available 
to assess uncertainty ? 

What models and tools are available 
for risk analysis? 


Requirements to apply risk methodology 


[PHASES in LIFE CYCLE 
ee eee arr EEE EEE _ _ —————— Eee SS 


ibility | Feasibility Project defini [Design and /Production 
phase phase phase |Development phase} phase iphase 
a 


Analysis Phase Procurement Phase | _—s—sServiceUse 


LCC analysis to compare alternatives and options, 
CBA 


LCC analysis to compare alternatives 
and options, 
CBA 


LCC analysis to compare alternatives 
and options, 
CBA 


CPM: approval required to enter next phase 


CPM: approval required to enter next 
phase 


CPM: approval required to enter next 
phase 


in process to start using FACET 


ACES 
PRICE 
CostXpert 
(COCOMO II) 


ACES 
PRICE 
CostXpert 
(COCOMO 11) 


ACES 
PRICE 
CostXpert 
(COCOMO Il) 


SCOPE 
Excel-sheets 


SCOPE 
Excel-sheets 


SCOPE 
Excel-sheets 


SCOPE 
Excel-sheets 


Expert opinion, Analogy, Parametric, Engineering 


Expert opinion, Analogy, Parametric, 
Engineering 


Expert opinion, Analogy, Parametric, 
Engineering 


Expert opinion, Analogy, Parametric, 
Engineering, Simulation 


mandatory 


mandatory 


mandatory 


mandatory 


none 


Availability of data, personnel, time and budget 


Manually, Information systems (limited), Answers to 
questionnaires 


none 
none 


Availability of data, personnel, time and 
budget 


Manually, Information systems (limited), 
Answers to questionnaires 


none 


none 


Availability of data, personnel, time and 
budget 
Manually, Information systems (limited), 
Answers to questionnaires 


none 
none 


Availability of data, personnel, time and 
budget 


Manually, Information systems (limited), 
Answers to questionnaires 


Expert opinion 
(IAGFA) 


Sensitivity Analysis 


Sensitivity Analysis 


Sensitivity Analysis 


none 


Cost Estimation, WU 
AZF-Plan / Milestones 
VOCON / Reviews 


Risk Analysis of parametric models 


Demonstrators, Simulations; 
Cost Estimation, WU 
AZF-Plan / Milestones 


Risk Analysis of parametric models 


Cost Estimation, WU 
AZF-Plan / Milestones 
VOCON / Reviews 


Risk Analysis of parametric models 


Cost Estimation, WU 
AZF-Plan / Milestones 
VOCON / Reviews 


mandatory as part of CPM 


mandatory as part of CPM 


mandatory as part of CPM 


mandatory as part of CPM 


Abbreviations 


Commercial Models 


“In-House” model 


ee Model. Methodol 
AZF-Plan= Arbeits-/ Zeit-/Finanzplan (Work, 
Time, Financial Plan) 


ACES = Advanced Cost Estimating Systems SCOPE = Cost Estimating Tool for Ships 


ICBA= Cost Benefit Analysis COCOMO II = Software Estimating Tool 


ICBS= Cost Breakdown Structure CostXpert = Software Cost Estimating Tool 


CPM= Customer Product Management 
IAGFA= Integr. Arbeitsgr. 
Fahigkeitsanalysen (Integrated Working 
Team Ability Analysis) 

\VOCON= Vorhabens Controlling (Integrated 
Controlling System) 


PRICE = Parametric Review of Information for Costing and Evaluation 


FACET = Family of Advanced Cost Estimating Tools 


IW U= Wirtschaftlichkeitsuntersuchung (Value} 
Research) 


* If you have products available that don’t need any modifications it is possible to reduce the Life Cycle of three phases (s. above) 


Figure A-3: Completed Matrix from Germany. 
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ANNEX A — NATION’S COMPLETED MATRIX 


ISSUE 01 
Date 13 Dec 2004 


ORGANIZATION 


PHASES in LIFE CYCLE 
a | ee | ee ee ee ee ee See 


NATO LCC Phases 


What methodologies are available ? 


Requirements to apply models/methodology 


Restrictions on applicable methods 
depending on the goal 


What models and tools are available 
ito assess uncertainty ? 


What models and tools are available 
for risk analysis? 


Requirements to apply risk methodology 


Abbreviations 


CBA = Cost Benefit Analysis 
CBS = Cost Breakdown Structure 
RFI = Request for Information 
IBD- IS TO BE DEFINED 


National LCC Phases A . : 


/Mission need |Pre-feasibility Feasibility | Design and ]Production In-service 
evaluation phase phase phase Project definition phase {Development phase phase phase Disposal phase 


CBA ,CBS 


Operational Analysis Studies 


RFI 


CBS Formulas 


Expert Analysis 
IBD 


IBD 


Expert Analysis 


User Requirements 


Manually 


Expert Analysis 


RFI 


Commercial Models “In House” Models 


Model 


Figure A-4: Completed Matrix from Greece. 
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ANNEX A — NATION’S COMPLETED MATRIX 


Issue PHASES in LIFE CYCLE 
EE. SE 


INATO LCC Phases 


National LCC Phases 
Nation ITALY 


Type of costing studies required 


Description of the process or national 
guideline to be followed - 
how are we going to do this ? 


What methods are used ? 

What models are available ? 
Commercial models 

What models are available ? 

In house developed models 
Requirements to apply national guideline 
Requirements to apply methods 


Requirements to apply models 


Restrictions on applicable methods or 
models depending on the goal 


How can data be collected 


VIISSION 
need 
evaluation 


Italian Army uses the NATO Life Cycle phases 


a . " TDSSIOIT and ; = 
feasibility | Feasibility | definition |Development [Production |In-service [Disposal 
phase phase phase |phase phase phase phase 


Overall 
constraints 
of costs 
and 
industrial 
capacity 


Not 
applicable 


Estimates of costs for 
prototypes and initial 
production, target unit 
production costs, life 
cycle costs, impact of 
inflation, budget 
changes, exchange 
rates 


RFP or equivalent, as applicable 


Update prior estimates as appropriate 


Specific 
costing 
studies 
(according 
to the 
nature of 
systems 
concerned) 


Analogy or parametric methodologies 


No specific commercial model 


Excel-based models and/or proprietary algorithms 


Some cost formats are mandated by the Government 


Some cost formats are mandated by the Government 


Some cost formats are mandated by the Government 


No particular restriction for methods/models 


Data collection is usually not automated 


How is risk and uncertainty considered ? 


No standard exists for risk and uncertainty evaluation 


What models and tools are available 
to assess uncertainty ? 


What models and tools are available 
for risk analysis? 


Requirement to apply risk methodolog 


None 


None 


None 


Figure A-5: Completed Matrix from Italy. 
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n 


National LCC Phases 
Nation Norway 
Type of costing studies required 


Description of the process or national 
guideline to be followed - 
how are we going to do this ? 


What methods are used ? 


What models are available ? 
[Commercial models 


What models are available ? 
In house developed models 


Requirements to apply national guideline 


Requirements to apply methods 


Requirements to apply models 
Restrictions on applicable methods or 
models depending on the goal 

How can data be collected 


How is risk and uncertainty considered ? 


What models and tools are available 
jto assess uncertainty ? 

What models and tools are available 
Hor risk analysis? 

Requirement to apply risk methodology 


issue I 
[Date 2006-09-10 _ ml 7 


v | Pr ion phase 
ee 


ant 


Development 


[ ql 
lprod 


nase | 


CBA 


CBA, WLC CBAWLC CBA.WLC CBA,WLG CBAWLC 
‘Operational Analysis Studies Feasibilly studies Feasbiliy studies Feasibility studies, follow-up Operational studies Feasibility studies 
Mandatory Mandatory RFI Mandatory RFQ Mandatory RFQ Mandatory Mandatory 
Analogies, Parametric, Expert opinion Activity Based Costing 
No model available or required opus 
IKOSTMOD, LCC-Analyzer, Xb 
Insgadl LOG ingle punnoes models KOSTMOD, LCC-Analyzer, XI-based LCC single purpose models, FLYT2 KOSTMOD, LCC-Analyzer LCC-Analyzer, 
National guideline exist for investments larger than 500 mill NOK (app 62 mill EUR) Significant divagation from decision basis needs to be approved No guideline on use of LCG 
Cost Benefit Analyste (Inel LEC) on Cost Benefit Analysis (incl LCC) on defined 
defined conseptual alternatives : ; : 
: alternatives within the decided consept is , ; ; 
mandatory to all investments with LCC. : : Significant divagation from decision basis needs to be approved 
ane re 200 mil NOK ape 62 mit | mandatory to all investments with LCC larger than 
ee ene (app 62 mi 500 mill NOK (app 62 mill EUR) 
EUR) 
No specific model requirements, but use of models are mandatory Desirable 
Data availability, knowledge, personnel, budget and time available 
Manually Manually Manually Manually Manually and information systems Manually 


Probability of project accomplishment. 


levaluated both in terms of: 
input and modelling and, 


Sensitivity analysis, expert opinion and 


lexperience on cost drivers, evaluation of risk and reliablity of input data based on alternative 
data sources and internal models i.e. related to logistics and availability rates. Risk should be 


1) risk in data 
2) risk in defined requirements for future 


capability (i.e. number of aircraft, new tehnology etc) 


Continuous risk evaluation in project using tools such as easy risk manager or similar. 


Easy Risk Manager, Extend, win RAMA, Orbit RCM, OPUS, @RISK, Crystal Ball, CARA fault tree, Access based FMECA tool 


Mandatory 


[Commercial Models 


CBA = Cost Benefit Analysis 
CBS = Cost Breakdown Structure 
COO = Cost of Ownership 

EVM = Earned Value Management 
IA = Investment Appraisal 

RFI = Request for Information 
RFQ = Request for Quotation 

SME = Subject Matter Expert 
UPC= Unit Production Cost 


@Risk = Risk analysis tool 
(Crystal Ball = Risk analysis tool 

OPUS = Modelling and spares optimisation 
Extend 

lwinRAMA 

Orbit RCM 

CARA fault tree 


Easy Risk Manager 
Access based FMECA tool 
LCC Analyzer 

ILSR data base 


KOSTMOD = Force Structure Cost 
FLYT2 = Air Force model, availability 


of aircraft and pilots 


Parametric 

Activity based costing 
Weighted marking scheme 
|Comparative/Parametric 
Risk based Parametric 
LCC Database 


Figure A-6: Completed Matrix from Norway. 


(ll 


ORGANIZATION 
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Issue 
Date 2006-04-20 
INATO LCC Phases: 


National LCC Phases 
Nation Sweden 


Type of costing studies required 
Description of the process or national 


guideline to be followed - 
how are we going to do this ? 


What methods are used ? 


|What models are available ? 
[Commercial models 


|What models are available ? 
In house developed models. 


Requirements to apply national guideline 
Requirements to apply methods 
Requirements to apply models 


Restrictions on applicable methods or models 
depending on the goal 


How can data be collected 
How is risk and uncertainty considered ? 


What models and tools are available 
to assess uncertainty ? 

What models and tools are available 
for risk analysis? 

Requirement to apply risk methodology 


ANNEX A — NATION’S COMPLETED MATRIX 


[Concept developement phase 


== ‘Project definition phase 7 
phase, Development phase 


[Concept evaluation phase 


Definition and demonstration phase 


[Production 


a cea 
phase J phase 


Procurment phase 


In-service phase 


Disposal phase 


[Costing for long term force structure planning. 


CBA 


CBA 


CBA 


[Operational Analysis Studies, not mandatory 


RFT, not mandatory 


RFT, RFP, not mandatory 
The Act on Public Procurement is mandatory 
IGate: A number of different cost analyses has to be performed 


before a project is allowed to proceed to the Production phase, 
e.g. An estimate of Life Cycle Cost. 


RFQ 


The Act on Public Procurement 
is mandatory 


Before a system is handed over 
to the Swedish Armed Forces 
trom FMV an estimate of 
loperating and maintenance 
Icosts shall be calculated by 
FMV. 

The estimate shall include 
scope, time plan for the outcome| 
lof costs divided on different 
resources and organisations and 
shall include cost for: 

loperating and maintenance, 
lconsumtion of maintenance 
resources, operating and 
maintenance management, 
technical support, ete 


RFQ 


The Act on Public Procurement 
is mandatory 


RFQ 


The Act on Public Procurement 
is mandatory 


CBS CBS CBS CBS 
Parametric Parametric Parametric Parametric 
lAnalogy Analogy Analogy Analytical 
Expert opinion Expert opinion Expert opinion Normorganisation to define a baseline for the LCC-calculations 
Rules of Thumb Sensitivity analysis to find cost drivers Sensitivity analysis to find cost drivers Discrete Event Simulation 
Normorganisation to define a baseline for the Normorganisation to define a baseline for the LCC-calculations Optimisation 
LCC-calculations Rules of Thumb 
Rules of Thumb Discrete Event Simulation 
Discrete Event Simulation Optimisation 
ICATLOC CATLOG ICATLOC 
SIMLOX SIMLOX 
lOPpus10 


[CBS-model (including calculation formulas) 


lExcelbased models 


Various ILS systems, 


]CBS-model (including calculation formulas) 


Excelbased models 


INo national guidelines on the use of LOC for any phase 
HThe Act on Public Procurement is mandatory 


in larger projects the Armed Forces requires various maintenance analyses and OBS 


[Some models are recommended 


ut none are required 


Data availability, knowledge, personnel, time, and budget 


From Business Software Applications, ILS-systems, Industry, Manually, Interviews/Questionaires, Expert opinion 


[Sensitivity analysis, Expert opinion 


Risk management plan, FMV working processes are linked to PMBOK Guide (A Guide to the Project Management Body of Knowledge) PMI - 


Earned value management 


Project Management Institute. Sensitivity analysis, Expert opinion, 


[Spreadsheet models, templates and calculation formulas 


[Spreadsheet models, templates and calculation formulas based on working process and methodology from (A guide fo the Management Body of Knowledge) PMI - 


processes. Earned value management is recommended. 


Project Management Institute and internal working 


[Spreadsheet models, templates and calculation formulas 


[Spreadsheet models, templates and calculation formulas based on working process and methodology from (A guide to the Management Body of Knowledge) PMI - 


processes. Earned value management is recommended. 


Project Management Institute and internal working 


The Swedish Armed Forces requires the Swedish Defence Materiel Administration to perform project risk analysis, no specific models are required except working processes. FMV Risk management is linked to FMV 
working processes for project management which are linked to PMBOK Guide (A Guide to the Project Management Body of Knowledge) PMI - 


management is also linked to ISO 15288:5.4.6 Risk Management Process. Earned value management is recommended. 


Project Management Institute. FMV working processes for Risk 


Abbreviations 


[Commercial Models: 


In House Models 


Model 


[Methodology 


RFI = Request for information 
RFQ = Request for Quotation 
RFP = Request for proposal 
CBA = Cost benefit analysis 
CBS = Cost Breakdown structure 
ILS = Integrated Logistic Support 


ICATLOG - system model for Life 
Cycle Cost (LCC) calculation 
land analysis 

OPUS - system model for 
modelling and optimization 
SIMLOX - discrete event Monte 
Carlo simulation mode! 


ASTOR - Air Force Simulation of 
Tactics and Operational 
Resources 


outstanding. 


The Act on Public Procurement - Since FMV is engaged in public sector procurement, there are rules and regulations 
that have to be followed. The principal regulations are contained in The Act on Public Procurement which applies to all 
public sector organisations. 

‘The fundamental principle is that procurement should be conducted in a businesslike, competitive and objective way. The 
regulations also contain rules governing procurement procedures, tender enquiries, the security of classified information, 
tender opening, assessment of tenderers, tender evaluation and other matters that must be followed by the procuring 
authority. FMV must also check that a new contractor is registered for VAT and does not have any tax payments 
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Figure A-7: Completed Matrix from Sweden. 
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ANNEX A — NATION’S COMPLETED MATRIX 


Issue [PHASES in LIFE CYCLE 
[Date SEES _ EESESESE——EEE—EE— — Ea 


INATO LCC Phases Mission need Pre-feasibility Feasibility Project definition Design and Production In-service 
I evaluatio! lase phase phase phase evelopment phase} phase | phase Disposal phase 
National LCC Phases Evaluation 
P rt i i 
Naten: oiuzecand ee ee Se ee ee avon SHO ee ee ii si 
é P Studies to Compare Strategic CBA CBA / WLC coo Operational Studies Feasibility Studies 
[Meceicosinisistse mice Alternatives, CBA Mandatory Desirable 


Description of the process or national 
guideline to be followed - Operational Analysis RFI Tender Assessment Contract Data Actual Costs Independent Estimate 
how are we going to do this ? Mandatory Independent Estimate 


What methods are used ? Analogy cBS CBS CBS cBS Analogy 

it models are availabl Excel Sheets Ecel Sheets Excel Sheets Ecel Sheets Ecel Sheets Ecel Sheets 
(Commercial models 

it ilal 


What 
SAP Business Warehouse SAP Business Warehouse SAP Business Warehouse usiness Warehouse SAP Business Warehouse SAP Business Warehouse 
Le ‘ Excel Sheets Excel Sheets Excel Sheets Excel Sheets Excel Sheets Excel Sheets 
In house developed models 
R f | quidel Desirable Mandatory for Major Projects Mandatory for Major Projects Desirable Desirable Desirable 
lequirements to apply national guideline Desirable Desirable 
Requirements to apply methods Desirable Desirable Desirable Desirable 


Is 
Requirements to apply models 
Is 


Desirable Desirable 


le ? 

models are available ? 

i Desirable Desirable Desirable Desirable Desirable Desirable 
Restrictions on applicable methods or Personal, Time, System & Budget Personal, Time, System & Budget 

models depending on the goal Requirements Requirements 

How can data be collected Mandatory Mandatory Mandatory 

How is risk and uncertainty considered ? None None None None None None 
What models and tools are available None None None None None None 
ito assess uncertainty ? 

What models and tools are available None None 

for risk analysis? 

ee eee rs Se baka re ee eee 


Model | Methodology 


Mandatory Mandatory Mandatory 


None None None None 


Abbreviations Commercial Models 


‘ost Breakdown Structure 


WLC = Whole Life Cycle Cost 


Figure: A-8: Completed Matrix from Switzerland. 
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ANNEX A — NATION’S COMPLETED MATRIX 


evaluation phase 


[PHASES in LIFE CYCLE 


[=] 
3 
:=] 
8 

FA 


Nation The Netherlands 


‘Type of costing studies required 


Description of the process or national 
guideline to be followed - 
how are we going to do this ? 


What methods are used ? 


Pre-Phase A 


Phase A: Statement of requirement 


Phase B/C/D: (preliminary) 
study & preparation for 
procurement 


High level studies to compare 
strategic alternatives 


Determine costs of currrently used equipment 


that need to be replaced (to be used as a benchmark) 


Preparing questionnaires 


LCC analysis to 
compare alternatives and 
options 
CBA and Business Cases 
Preparing questionnaires 


SSS SSS] 


LCC analysis to 
compare alternatives and 
options 
CBA and Business Cases 
Preparing questionnaires 


In-service 


LCC analysis to 
compare alternatives and 
options 
CBA and Business Cases 
Preparing questionnaires 


Disposal phase 


Disposal 


Replacement studies, CBA 


DMP /RFI: approval required to enter next phase, 


DMP / RFQ & RFP 
approval required to enter 


FEL-SALDO step-by step 
approach for LCC-analysis 


DMP Phase E: evaluation of 
process followed (only large 


next phase projects) 
FEL-SALDO step-by step approach for LCC-analysis FEL-SALDO step-by step 
approach for LCC-analysis 
Analogy Analogy, Engineering, Analogy, Engineering, Engineering Rule of Thumb 
Analogy, Parametric Parametric, Expert opinion, | Parametric, Expert opinion, 
Rule of Thumb Rule of Thumb 
none none none none none 


none 


LCC tool and Excel 
spreadsheet tools 


LCC tool and Excel spreadsheet tools 


LCC tool and Excel 
spreadsheet tools 


LCC tool and Excel 
spreadsheet tools 


LCC tool and Excel 
spreadsheet tools 


LCC tool and Excel 
spreadsheet tools 


DMP process is obliged 


DMP process is obliged, 
FELSALDO CBS is obliged in DMP 


DMP process is obliged, 
FELSALDO CBS is obliged in 


DMP process is obliged, 
FELSALDO CBS is obliged in 


DMIP is under development 


DMP DMP 
none none none none none 
none 
none fore none none none none 


Restrictions on applicable methods or models 
depending on the goal 


Availability of data, personnel,] 
time and budget 


Availability of data, personnel, time and budget 


Availability of data, personnel, 
time and budget 


Availability of data, personnel, 
time and budget 


Availability of data, personnel, 
time and budget 


Availability of data, personnel, 
time and budget 


How can data be collected 


Manually, Information 
systems (limited), Answers to 
questionnaires 


Manually, Information systems (limited), Answers to questionnaires 


Manually, Information 
systems (limited), Answers to 
questionnaires 


Manually, Information 
systems (limited), Answers to 
questionnaires 


Manually, Information 
systems (limited), Answers to 
questionnaires 


Manually, Information 
systems (limited), Answers to 
questionnaires 


How is risk and uncertainty considered ? 


sensitivity analysis, 
expert opinion 


sensitivity analysis, 
expert opinion 


sensitivity analysis, 
expert opinion 


sensitivity analysis, 
expert opinion 


sensitivity analysis, 
expert opinion 


sensitivity analysis, 
expert opinion 


What models and tools are available 
to assess uncertainty ? 


New (in house developed) 
LCC tool covers uncertainty, 
Crystal Ball is also available 


New (in house developed) LCC tool covers uncertainty, 


Crystal Ball is also available 


New (in house developed) 
LCC tool covers uncertainty, 
Crystal Ball is also available 


New (in house developed) 
LCC tool covers uncertainty, 
Crystal Ball is also available 


New (in house developed) 
LCC tool covers uncertainty, 
Crystal Ball is also available 


New (in house developed) 
LCC tool covers uncertainty, 
Crystal Ball is also available 


What models and tools are available 
for risk analysis? 


Requirement to apply risk methodology 


mandatory as part of DMP 


mandatory as part of DMP 


mandatory as part of DMP 


|Abbreviations 


LCC = Life Cycle Costs 
CBS = Cost Breakdown Structure 


CBA = Cost Benefit Analysis 

FELSALDO = FEL Step by step Approach 
to Analyse Life cycle costs in the Defence 
Organisation 


DMP = Defence Materiel Process 

RFI = Request for Information 

RFQ = Request for Quotation 

RFP = Request for Proposal 

DMIP = Defence Materiel In-service Process 
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Commercial Models 


In House" Models 


Model 


Methodology 


Crystal Ball 


LCC tool 


Figure A-9: Completed Matrix from The Netherlands. 


Type of costing studies required 


ANNEX A — NATION’S COMPLETED MATRIX 


Pre-concept 


CBA 


Concept 


CBA, UPC, WLC 


Project Definition Phase 


Engineering & Product 
Development 


UPC, WLC 


Manufacture 


Operation & Maintenance 


WLC 


Disposal 


ORGANIZATION 


Description of the process or national 
guideline to be followed - 
how are we going to do this ? 


OR Studies, Construction of CBS 


Elaboration of CBS, RFI, RFP 


‘Tender Assessment, Scheduling 


Verification of CBS, 
Tender Assessment 


Verification of CBS, Scheduling 


Verification of CBS, 
Deriving actual costs 


What methods are used ? 


Analogy, Contingency Analysis, Pairwise Comparisons, 
Multi-objective Analyses 


Parametric Comparisons, 
Regression Analysis 


Parametric Analyses 


Parametric Analyses, 
Discrete Event Simulation 


Discrete Event Simulation 


Activitiy Based Costing 


Parametric comparisons 


What models are available ? 
(Commercial models 


EXCEL, ExpertChoice, Theater Level Simulation Tools 


EXCEL, MS Project Manager 


EXCEL, MS Project Manager 


EXCEL, ARENA 


EXCEL, ARENA, 
MS Project Manager 


EXCEL, ARENA 


EXCEL, Expert Choice 


What models are available ? 
In house developed models 


BESTSEL, BAM 


GVCAM, FMSCAM, ALCAM, 
BAM 


ALCAM, P3M 


CALS* 


CALS* 


CALS*, EDCAM, EXCAM 


CALS*, BESTSEL 


Requirements to apply national guideline 


Desirable 


Desirable 


Mandated 


Mandated 


Mandated 


Mandated 


Desirable 


Requirements to apply methods 


Desirable 


Desirable 


Desirable 


Desirable 


Desirable 


Some methods are mandated 
by the General Staff 


Desirable 


Requirements to apply models 


Desirable 


Desirable 


Desirable 


Desirable 


Desirable 


Some models are mandated by 
the General Staff 


Desirable 


Restrictions on applicable methods 
depending on the goal 


How can data be collected 


Personnel, time, system and budget requirements 


Manual Records, General Information & Data Systems 


Personnel, time, system and 
budget requirements 
Manual Records, General 
Information & Data Systems 


Manual Records, General 
Information & Data Systems 


Manual Records, General 
Information & Data Systems 


Manual Records, General 
Information & Data Systems 


Manual Records, General 
Information & Data Systems 


Manual Records, General 
Information & Data Systems 


How is risk and uncertainty measured ? 


Sensitivity Analysis 


Sensitivity Analysis, 
Monte Carlo Simulation 


Monte Carlo Simulation 


Monte Carlo Simulation 


What models and tools are available 
ito assess uncertainty ? 


What models and tools are available 
for risk analysis? 


CALS", RISKSIM, Statistical Software Packages 


CALS", RISKSIM, Statistical 
Software Packages 


CALS’, RISKSIM, Statistical 
Software Packages 


CALS’, RISKSIM, Statistical 
Software Packages 


Requirements to apply risk methodology 


CBA = Cost Benefit Analysis 
CBS = Cost Breakdown Structure 
RFI = Request for Information 
RFQ = Request for Quotation 
SME = Subject Matter Expert 
UPC= Unit Production Cost 
WLC= Whole Life Cost 


Access to SME 


‘Commercial Models 


[ARENA Standard Edition 8. 


Optimization Tools : 


0 (including OptQuest feature) 


Expert Choice (Advanced Desicion Support Software) 
Theater Level Simulation Tools : NIMROD, TAMARI 
‘Statistical Software Packages : JMP, SPSS, Bestfit 
Lingo 8.0 Extended Optimizasyon Paketi 


Extended What's Best 7.0 
Extended Large-Scale LP 
Solver Engine V5.5E 


Premium Solver Platform for Excel V5.5 


General Information & Data 
Systems : 


BESTSEL : Best Selection (Prioritization Tool Based on Optimization) 
BAM : Budget Allocation Model 

GVCAM : Ground Vehicle Cost Analysis Model 

ALCAM : Aircraft Life Cycle Cost Analysis Model 
FMSCANM : Foreign Military Students Cost Analysis Model 
P3M : Pre-payment for Projects Model 

EDCAM : Educational Cost Effectiveness Analysis Model 
EXCAM : Exercises Cost Effectiveness Analysis Model 
CALS - ILS : Continuous Acquisition Lifetime Support - Integrated Logistic Support Module 
Air Forces Information System, 
Logistics Information System 
C2 Information System 


Sensitivity Analysis 
Contingency Analysis 
Heuristics 

‘Scheduling Theory 
Regression Analysis 
Activity Based Costing 


Figure A-10: Completed Matrix from Turkey. 
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ANNEX A — NATION’S COMPLETED MATRIX 


SSUE 02 PHA’ T YCLE 
EE Ee a ae Saas aaa: aki 3 
[Mission need Pre-feasibility Feasibility | Project definition |Design and Production in-service |Disposal 
jevaluation phase_| phase phase _| phase [Development phase se lphase phase 
Pre System Acquisition System Acquisition Sustainment 
National LCC Phases 
Nation: U.S. Dept. of Defense Concept Refinement Technology Development poem paves & ooo : ee ee in 
DoD Instruction 5000.2, Issued May emonstenon epioymes) uPPO! 
2003 Possible additional milestones for 
Milestone A 4] Milestone B Milestone C ‘| major updates 


At milestone B and C: AF; APB; CARD for MDAPs & ICEs for FRP DRs (in production 
MAIS; cost assessment (for certain ACAT levels); phase); also, updates of prior cost 
LCCE; EA & CCA for MAISs; ICE for MDAPs; PBL EA estimates for new increments 


e : ‘ "Rough cost assessment" as part 
Type of costing studies required of JCIDS analysis; AoAs 


Description of the process or national 


Buidelinettolbe alowed The U.S. DoD cost analysis community follows policies and procedures as promulgated in DoD Instruction 5000.2, "Operation of 


the Defense Acquisition System," and in all implementing guidance to this instruction. 


Rough analogies; high-level Analogy, parametric |Parametric & Analogy & 
What methods are used ? parametric cost estimating Analogy & Parametric & engineering build jengineering |Parametric Patani 
relationships (CERs) up build up 


What models are available ? 
‘Commercial models 

What models are available ? 
In house developed models. 


Literally hundreds of models are available. Each DoD component involved in cost estimating buys and develops tools and models 
appropriate to the types of systems relevant to that organization. MS Excel is the DoD standard spreadsheet. For MDAPs and 
MAISs, OSD and each Service typically develop a cost-estimating model unique to that system. 


Requirements to apply national guideline 

DoD and Service-level guidance specify, in detail, what cost-analysis products are required at various milestones in the acquisition 

process. However, considerable latitude is permitted in terms of how to actually produce the deliverables. The individual analyst 
has great intellectual freedom in generating a cost estimate. 


Requirements to apply methods 
Requirements to apply models 
Restrictions on applicable methods or 
models depending on the goal 


Cost data are available through the DoD and Service budgeting and accounting systems; formal reports 
How can data be collected such as CCDRs, SRDRs, CPRs, SARs, DAES reports; internal contractor accounting data; and through | VAMOSC | VAMOSG. 
numerous cost/technical database efforts throughout the cost community. 


How is risk and uncertainty considered ? Varies within and between cost-estimating organizations. A detailed study is available. 


What models and tools are available 
to assess uncertainty ? 

What models and tools are available 
for risk analysis? 


Varies by cost-estimating organization. A detailed study is available. 


Commercial tools such as Crystal Ball and @Risk are widely available in DoD but not used consistently. 


Requirement to apply risk methodology Varies by cost-estimating organization. A detailed study is available. 


List of abbreviations| 


ACAT|Acquisition Category 
AF| Affordability Assessment 
AoA] Analysis of Alternatives 
APB|Acquisition Program Baseline (contains cost goals, in addition to performance and schedule goals) 
CARD] Cost Analysis Requirements Description 
CCA|Component Cost Analysis 
CCDR|Contractor Cost Data Report 
CER|Cost Estimating Relationship 
CPR|Contract Performance Report 
DAES|Defense Acquisition Executive Summary report 
DoD] Department of Defense 
EA|Economic Analysis 
FRP DR|Full-rate Production Decision Review 
ICE| Independent Cost Estimate 
JCIDS| Joint Capabilities Integration and Development System (CJCSI 3170.01E; 11 May 2005) 
LCCE]Life Cycle Cost Estimate 
MAIS|Major Automated Information System 
MDAP|Major Defense Acquisition Program 
PBL|Performance Based Logistics 
SAR|Selected Acquisition Report 
SRDR|Software Resources Data Report 
VAMOSC|Visibility and Management of Operating and Support Costs 


oO 


Figure A-11: Completed Matrix from the United States of America. 
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ANNEX A — NATION’S COMPLETED MATRIX 


A- 12 


ISSUE 03 
Date 04 April 2006 


|NATO LCC Phases 


National LCC Phases 
Nation UK MOD CADMID 


Type of costing studies required 


Description of the process or national 
guideline to be followed - 
how are we going to do this ? 


What methods are used ? 


What models are available ? 
(Commercial Models 


What models are available ? 
In House Models 


Requirements to apply national guideline on 


LCC 


Requirements to apply methods 
Requirements to apply models 


Restrictions on applicable methods and/or 
models depending on the goal 


How can data be collected ? 
How is risk and uncertainty measured ? 


What models and tools are available 
to assess uncertainty ? 


What models and tools are available 
for risk analysis ? 


Requirements to apply risk methodology 


[PHASES in LIFE CYCLE 
[Mission need 
levaluation phase 


Pre-feasibility 
phase 


Pre-Concept 


Bol, CBA 


UPC, WLC, COO 


—SSeeeeS SSS ee eee aa SSeS 
Feasibility = Design and Production ‘service | 
phase Project definition phase _ {Development phase | phase phase [Disposal phase 


peeenen Demonstration Manufacture Pisposal 


UPC,IA, WLC, COO 


UPC,IA, WLC, COO 


UPC,WLC, COO 


UPC, WLC, COO 


IA, WLC, COO 


WLC, COO 


Operational Analysis Studies 


Bayesian, Parametric, 
Analogy 


RFI, Independent estimate 


RFQ, Independent estimate 


Tender Assessment 


Contract data 


Actual Costs 


Independent estimate 


Saaty, Pair-Wise, 


Bayesian, Parametric, CGT, 
Regression Analysis 


CBS, CGT,Regession 
Analysis, COO 


CBS, CGT,Regession 
Analysis, COO 


CBS, CGT, MACE, COO 


CBS, COO 


CBS, COO 


Analogy, COO 


Portfolio, Option Analyser 


Operational Analysis Studies 


FACET, PRICE, SEER 


EXCEL,SPRUCE, MELICCA, 
A-credit, OATS, SSCM 


FACET, PRICE, SEER, 
COCOMO 


EXCEL,SPRUCE, MELICCA, 
A-credit, OATS & COO 
System, SSCM 


FACET, PRICE, SEER, 
COCOMO, OSCAM, ILS 
Tools 
EXCEL,SPRUCE, MELICCA, 
A-credit, OATS & COO 
System, SSCM 


ILS Tools, OSCAM, 
COCOMO 


EXCEL,SPRUCE, MELICCA, 
A-credit, OATS & COO 
System, SSCM 


Metify, OSCAM, OPUS 


Excel, OATS & COO System 


Metify, OSCAM, OPUS 


Excel, OATS & COO System 


Metify, OSCAM, Excel 


Excel, OATS & COO System 


Desirable Desirable Mandatory for major Projects | Mandatory for major Projects | Mandatory for major Projects | Mandatory for major Projects | Mandatory for major Projects / Desirable 
/ Desirable / Desirable / Desirable / Desirable Desirable 

Desirable Desirable Desirable Desirable Desirable Desirable Desirable Desirable 

Desirable Desirable Desirable Desirable Desirable Desirable Desirable Desirable 


Access to SME, User 
Requirements 


Access to SME, User 
Requirement 


Access to Data, System 
Requirement 


Access to Data, Design Data 


Access to Data, Design Data 


Contract conditions 
restricting access to actuals, 


Contract conditions restricting 
access to actuals 


Planning Assumptions 


Design data 
Manually Manually Manually Manually Manually Manually, EVM Manually Manually 
Sensitivity Analysis Sensitivity Analysis Detailed Modelling Detailed Modelling Risk Management & Analysis | Risk Management & Analysis| Risk Management & Analysis Sensitivity Analysis 


@RISK, Predict, Arrisca, 
Crystal Ball, Optimism Bias 


G@RISK, Predict, Arrisca, 
Crystal Ball, Optimism Bias 


GRISK, Predict, Arrisca, 
Crystal Ball, Optimism Bias 


GRISK, Predict, Arrisca, 
Crystal Ball 


@RISK, Predict, Arrisca, 
Crystal Ball 


GRISK, Predict, Arrisca, 
Crystal Ball 


@RISK, Predict, Arrisca, 
Crystal Ball 


GRISK, Predict, Arrisca, 
Crystal Ball 


[Abbreviations 


CBA = Cost Benefit Analysis 

CBS = Cost Breakdown Structure 
COO = Cost of Ownership 

BOI = Balance of Investment 

EVM = Earned Value Management 
[A = Investment Appraisal 

RFI = Request for Information 
RFQ = Request for Quotation 

SME = Subject Matter Expert 
UPC= Unit Production Cost 


TDRM TDRM ARM, Arrisca, Predict Risk | ARM, Arrisca, Predict Risk | ARM, Arrisca, Predict Risk | ARM, Arrisca, Predict Risk ARM, Arrisca, Predict Risk TDRM, PERTMaster 
Controller, PERTMaster Controller, PERTMaster Controller, PERTMaster Controller, PERTMaster Controller, PERTMaster 
Mandatory Mandatory Mandatory Mandatory Mandatory Mandatory Mandatory 
(Commercial Models "in House” Models i 
Model. Methodology 
@Risk = Risk analysis tool CGT = Compensated Gross Tonnage (Ships) Parametric 


|A-Credit = Automated cost resource evaluation and data integration tool 


ARM = Active Risk Manager 


Arrisca = Risk Management and analysis tool (Cost & Schedule) 


Force Structure Cost Model 


MACE = ‘Multi(ple) Attribute Choice Elucidation’ (Cost is just one element) 
MAGRO (Method for Assessing Cost of exploiting Research Output) 


ICOCOMO = Software estimating tool 

Crystal Ball = Risk analysis tool 

FACET = Family of Cost Estimating Tools 

/Metify = Activity based costing/management (ABC/M) 

OPUS = Modelling and spares optimisation 

PERT Master = Schedule risk management and analysis tool 

Predict Risk Controller = Schedule risk management and analysis tool 
PRICE = Parametric cost estimating models 

SEER Parametric cost estimating models 


SSCM = Software Support Cost model 
TDRM = Top Down Risk Model 


IMELICCA = Cost collection tool 


SPRUCE: Ship Platform Risk based Unit Cost Estimates 


OATS & COO System = Options and Affordability Tools Set & Cost of Ownership System 


Activity based costing 
Weighted marking scheme 
Comparative/Parametric 
Activity based costing 

Risk based Parametric 
Activity based costing 


Figure A-12: Completed Matrix from the United Kingdom. 
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ORGANIZATION 
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ORGANIZATION 


Annex B —- EXAMPLES OF DATA FORMS FOR CAPTURING 
THE COSTS AT THE PRODUCTION PHASE 


Sheet 1 of 2 Attachment to PFG/PR/ 208: 
Appendix A 


CONTRACTOR DATA PACK AVAILABILITY Issue 1 
Insert MoD Proposal Reference here please > 


For information refer to; MOD Guidelines to Industry "Partnering between MOD and its Suppliers" 


BELOW IS A LIST OF GENERIC ITEMS 
TICK THE RELEVANT COLUMN TO INDICATE IF THE ITEM IS AVAILABLE TO SUPPORT YOUR QUOTATION 


IF YES - QUOTE REF. and ISSUE 
ITEMS INCLUDED IF NO - please comment 


Preferred, all items below on 3.5" disk(s), except items 9, 14 to 29. Available 
MS Office Word 7.0a for Text Files, Excel 7.0a for Spreadsheets on Disc? 


Work Breakdown Structure Details 


List of Work Packages 
Listing of Cost centres used by depart./Divisions/Sites involved 


Risk Analysis Documentation (e.g. Risk Register) 


Specify Reference and Issue for quotation _—_} | —______________] 


(a) Build Standard 
(b) Technical Specification 
(c) Performance Specification 


(d) Specification of Requirements 

(e) Delivery Programme 

(f) Questionnaire on the Method of Allocation of Costs (QMAC) 
used as a base 


Development Cost Plan 


Production Cost Plan 


Unit Production Cost Estimates 


Parallel manuf. and Productivity Improvements/Efficiency Gains 


Payment Plan 
Estimating Rationale Statement 


List of estimating Allowances and Contingencies used, incl. 
Learner, escalation and contingency rationale 


Applied Estimating Assumptions 


Make/Buy Plan 


Material Estimates, 
(a) including rationale for going Competitive 
(b) rationale for Sub-contractor selection 


Figure B-1: UK Contractor Data Sheet — Appendix A. 
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ANNEX B — EXAMPLES OF DATA FORMS FOR = 
CAPTURING THE COSTS AT THE PRODUCTION PHASE oon 


Sheet 2 of 2 Attachment to PFG/PR/ 2082 

Appendix A 

CONTRACTOR DATA PACK AVAILABILITY Issue 1 
Insert MoD Proposal Reference here please >>> 


For information refer to; MOD Guidelines to Industry "Partnering between MOD and its Suppliers" 


BELOW IS A LIST OF GENERIC ITEMS 
TICK THE RELEVANT COLUMN TO INDICATE IF THE ITEM IS AVAILABLE TO SUPPORT YOUR QUOTATION 
IF YES - QUOTE REF. and ISSUE 


ITEMS | «YE IF'NO- please comment 


Yes 


List all Non-competitive Major Sub-contractors, 
(a) are supply copies of Quotations available 


List of all Proprietary Items and Basis of Pricing, linked to WPs. 


Labour Estimates 


Statement defining any New or innovative manuf. techniques 


List assumptions for all Government Funded Equipment (GFE) 
List proposed Tooling, including scale and scope 


Statement on the Economic Datum Point of Estimate/Quotation, 
ie. June 2001 


Inflation Factors used and Source, e.g. RPI, Indices etc.. 


Labour and Material Spend Profile 


Recorded Costs 
(a) incurred against this Proposal, if converted to a contract 
(b) Previous Contracts for Same Equipment 
(c) Contracts of Similar Nature 


Details of the proposed Earned Value Management scheme (if 
applicable) and outputs from similar projects either completed 
or in progress. 


Direct Labour Rates Used and their Status. This should include 
the estimating assumptions regarding; 
(a) labour utilisation 
(b) future work programme covering the 
duration of the subject contract 
(c) department loadings 
(d) estimated % of MoD/Commercial workload split for the 
duration of the subject contract 


Assumptions regarding incentivisation and or gainshare. 


A statement regarding any omissions from the proposal or 

non-compliance with the RFQ/ITT eg. 

a. cannot provide a firm price proposal 

b. can only supply 75 instead of 100, in the delivery period 
requested. 


Figure B-1: UK Contractor Data Sheet — Appendix A (Continued). 
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ANNEX B — EXAMPLES OF DATA FORMS FOR 
CAPTURING THE COSTS AT THE PRODUCTION PHASE 


Attachment to PFG/PR/ 2082 
Appendix B 


CONTRACTOR DATA PACK - SUPPORT CHECKLIST 
Insert MOD Proposal Reference here please >>> 


For information refer to; MOD Guidelines to Industry "Partnering between MOD and its Suppliers" 


TICK THE RELEVANT COLUMN TO INDICATE THAT THE FOLLOWING ITEMS OR EQUIVALENT ARE AVAILABLE 


ON-SITE, ON REQUEST FOR MoD PFG 


IF YES - QUOTE REF. and ISSUE 


ITEMS AVAILABLE IF NO - please comment 


Quotation Specifications and Plans 
(i) Specification of Requirement linked to Task Booking Structure 
(ii) Build Standard/Technical Specification 
(iii) Performance Specification 
(iv) Quality Plan 
(v) Make/Buy Plan 
(vi) Delivery Programme 
Process Layout 
Process Specifications 
Operation Schedules 
Statistics to support any applied Learner Allowance 
Route Cards 
Manufacturing Drawings 
Parts Lists 
Test Specifications/Schedules 
Process Drawings/Specifications 
Competitive material and sub-contract quotations/invoices 
Statement on Plant, Processes and Techniques to be employed. 
Overheads Rates 
(i) Overheads Budgets/Forecasts 


(ii) Forward Load in Hours/Manning Levels 
(iii) Company Strategy and Rationalisation Plans 


| ves | NO 
a 
a a re 
PE 
PP 
PP 
a a 
es A 
PP 
ae a) ee 
PP 
PE 
PP 
PP 
EE 
PP 
PE 
PE 
ss eS 
PE 
a ee 
PE 
a es 
PE 
PP 
a Ce 
eS 
PP 
PE 


(iv) Pay Deals 


(v) Productivity Deals 


Copy of Business Plan 


Cash Flow & Profit and Loss Predictions 


Supporting Data for Claimed Profit Rate, including CP:CE Claims 


Access to Company Estimating Request Forms (ERFs) from 
Departmental bid returns. 


Access to Company Estimating Database information and/or 
Computer Models where appropriate. 


Figure B-2: UK Contractor Data Sheet — Appendix B. 
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ANNEX B — EXAMPLES OF DATA FORMS FOR 
CAPTURING THE COSTS AT THE PRODUCTION PHASE a 


illll 


SECURITY CLASSIFICATION 


Form Approved 
FUNCTIONAL COST-HOUR AND PROGRESS CURVE REPORT OMB No. 0704- 
0188 


The public reporting burden for this collection of information is estimated to average 45 hours per response, including the time for reviewing instructions, searching existing dala sources, gathering and maintaining the data needed, and completing and 
reviewing the collection of information. Send comments regarding this burden estimate or any other aspect of this collection of information, including suggestions reducing the burden to Department of Defense, Washington Headquarters Service, Directorate| 
for information Operations and Reports (0704-0188), 1251 Jeffereson Davis Highway, Suite 1204, Arlington, VA 22202-4302. Respondents should be aware that notwithstanding any other provision of law, no person shall be subject to any penalty for failing 
to comply with a collection of information if it does not display a currently valid OMB control number. 

PLEASE DO NOT RETURN YOUR FORM TO THIS ADDRESS 


1a. PROGRAM 1b. APPROVED PLAN NUMBER 2. REPORT AS OF (MM/DD/YY) 3. FY FUNDED 


4a. CONTRACTOR TYPE 5. DOLLARS IN 6. HOURS IN 
[__]PRIME/ASSOCIATE [__]susconTractor [__]SUBCONTRACT (Estimate by Reporting Contractor) 


4 b. NAME/ADDRESS (include ZIP Code) 


7a. CUSTOMER (Subcontractors Use Only) 7b. SUBCONTRACTOR (Estimated by Reporting Contractor) 8. SUBCONTRACT NO. 


9. NUMBER OF REPORTING SUBCONTRACTORS 
Cc] LATEST AMENDMENT 


11. MULTFYEAR CONTRACT 
Clves Jno | _]PROGRAM ESTIMATE 
12. WBS ELEMENT CODE PART |. FUNCTIONAL COST-HOUR REPORT 
14. COST TYPE 15. QUANTITY 16. APPROPRIATION 
[_]RECURRING TO DATE 
13. REPORTING ELEMENT | _JNONRECURRING JATCOMPLETION [sd [_|PROCUREMENT 


SUBCONTRACT OR OUTSIDE 
REPORTING CONTRACTOR PRODUCTION AND SERVICES TOTAL 


DATA ELEMENTS TO DATE AT COMPLETION TO DATE AT COMPLETION TO DATE AT COMPLETION 
A B Cc D E 


1. DIRECT LABOR HOURS 


12. DIRECT LABOR DOLLARS 

3. OVERHEAD 

4. MATERIAL 

5. OTHER DIRECT CHARGES (Specify) 

8. DIRECT LABOR DOLLARS. 

10. MATERIAL AND PURCHASED TOOLS 

11. OTHER DIRECT CHARGES (Specify) 

14. DIRECT LABOR DOLLARS 

16. OTHER DIRECT CHARGES (Specify) 
OTAL QUA Y CONTROL DOLLAR 

120. OVERHEAD 

21. MATERIALS AND PURCHASED PARTS 

22. OTHER DIRECT CHARGES (Specify) 
OTAL MANUFA URIN DO 

OTHER COSTS 


24, PURCHASED EQUIPMENT 
25. MATERIAL OVERHEAD 
6. OTHER COSTS NOT SHOWN ELSEWHERE (Speciy) 


UMMARY 
27. TOTAL COST (Direct and Overhead) 


28. REMARKS 


POINT OF CONTACT (POC) INFORMATION 


29a. NAME (Last, First, Middle Initial) 29b. DEPARTMENT l29c. TELEPHONE NO. (Include Area Code) 
29d. E-MAIL ADDRESS 29e. FAX NO. (Include Area Code) 29f. SIGNATURE 29g. DATE SIGNED (MM/DD/YY) 


DD FORM 1921-1, (FRONT), OCT 2003 


SECURITY CLASSIFICATION 


Figure B-3: USA Functional Cost Hour and Progress Curve Report (DD Form 1921-1 Front). 
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7) 7: ANNEX B — EXAMPLES OF DATA FORMS FOR 
CAPTURING THE COSTS AT THE PRODUCTION PHASE 


SECURITY CLASSIFICATION 
FUNCTIONAL COST-HOUR AND PROGRESS CURVE REPORT 


PART Il. PROGRESS CURVE REPORT 
T. WBS ELEMENT CODE : 

[_JuNrt ToTAL or [__]-oT TOTAL oR 
2. REPORTING ELEMENT. [_JUNrt AveRAGE [_]LOT AVERAGE 


COMPLETED UNITS/LOTS WORK-IN- 
eae aaa 
Cc 


DATA ELEMENTS 
. MODEL AND SERIES 
. FIRST UNIT OF LOT/WIP UNITS 


7 a) ) Ke) Es Ed 
ojo} 
alc 
lolP4 
Z| 
ala 
qc 
DZ 
a 
[e} 
Zo 
= 
Slo 
24 
=| 
gi 
| 
fe) 
+ 
n 


HARACTERISTICS 


ONTRA 


6. DIRECT QUALITY CONTROL LABOR HOURS 

7. DIRECT MANUFACTURING LABOR HOURS 

8. TOTAL DIRECT LABOR HOURS 

9. DIRECT QUALITY CONTROL LABOR DOLLARS 

T MANUFACTURING LABOR DOLLARS 
TOTAL DIRECT LABOR DOLLARS 

RAW MATERIALS AND PURCHASED PARTS 
PURCHASED EQUIPMENT 

TAL DIRECT DOLLARS 
SUBCONTRACT/OUTSIDE PRODUCTS AND SERVICES 
. DIRECT QUALITY CONTROL LABOR HOURS 
DIRECT MANUFACTURING LABOR HOURS 

TOTAL DIRECT LABOR HOURS 

DIRE UALITY CONTROL LABOR DOLLARS 

. DIRECT MANUFACTURING LABOR DOLLARS 

TAL DIRECT LABOR DOLLARS 

PURCHASED PARTS 

IASED EQUIPMENT 

DIRECT DOLLARS 

R UN O 
T QUALITY CONTROL LABOR HOURS 

| MANUFACTURING LABOR HOURS 

DIRECT LABOR HOURS 

T QUALITY CONTROL LABOR DOLLARS 

T MANUFACTURING LABOR DOLLARS 

RECT DIRECT LABOR DOLLARS 

PURCHASED PARTS 

IASED EQUIPMENT 

DIRECT DOLLARS 

33. % SUBCONTRACT OR OUTSIDE PRODUCTION AND SERVICES 


. REMARKS 


= 
Be) 
m 
QO 


Niro 


ALO] 
4 
g 


IN I< 
his) 
2) 


POINT OF CONTACT (POC) INFORMATION 


35a. NAME (Last, First, Middle Initial) 35b. DEPARTMENT 35c. TELEPHONE NO. (Include Area Code) 
35d. E-MAIL ADDRESS 35e. FAX NO. (Include Area Code) 35f. SIGNATURE 35g. DATE SIGNED (MM/DD/YY) 


DD FORM 1921-1, (BACK), OCT 2003 


SECURITY CLASSIFICATION 


Figure B-4: USA Functional Cost Hour and Progress Curve Report (DD Form 1921-1 Back). 
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ANNEX B — EXAMPLES OF DATA FORMS FOR 
CAPTURING THE COSTS AT THE PRODUCTION PHASE a 


illll 


Form Approved 


OMB No. 
SECURITY CLASSIFICATION 0704-0188 


3. TYPE ACTION 
COST DATA SUMMARY REPORT | |CONTRACT NO: 


ta. PROGRAM: 


7. FY FUNDED: 
1b. APPROVED PLAN NUMBER: 


12. NAME OF CUSTOMER: (Subcontractor Use Only) 
| _|PRIME/ASSOCIATE 
| _|SUBCONTRACTOR 
CONTRACT NUMBER 
LINE COSTS INCURRED OF COSTS INCURRED 
NONRECURRING RECURRING UNITS NONRECURRING | RECURRING 
E I H | 


13. REMARKS: 


POINT OF CONTACT (POC) INFORMATION 


14a. NAME (Last, First, Middle Initial) 14b. DEPARTMENT 4c. TELEPHONE NO. (Include Area Code) 
14d. E-MAIL ADDRESS 14e. FAX NO. (Include Area Code) 14f. SIGNATURE 14g. DATE SIGNED (MM/DD/YY) 


DD FORM 1921, OCT 2003 


Public reporting burden for this collection of information is estimated to average 33 hours per response, including the time for reviewing instructions, searching existing data sources, gathering and 
maintaining the data needed, and completing and reviewing the collection of information. Send comments regarding this burden estimate or any other aspect of this collection of information, 
including suggestions for reducing this burden, to Washington Headquarters Services, Directorate for information Operations and reports, 1215 Jefferson Davis Highway, Suite 1204, Arlington, VA 
22202-4302, and to the Office of Management and Budget, Paperwork Reduction Project (0704-0188), Washington, D.C. 20503. 
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Figure B-5: USA Cost Data Summary Report (DD Form 1921). 
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Software Resources Data Report: Initial Government Report - Sample 
Due 180 Days Before Contract Award as part of Cost Analysis Requirements Description (CARD) 
Page 1: Report Context, Project Description and 


Report Context 


1. System/Element Name (version/release): 2. Report As Of: 


3. Authorizing Vehicle (MOU, contract/amendment, etc.): 4. Reporting Event: CARD 


Submission # 


(Supersedes #___, if applicable) 


ee 
CO 


19. Secondary Language (expected or required): je. 


21. List COTS/GOTS Applications (expected or required): 


22. Peak staff (team size in FTE) expected to work on and charge to this project: 


Comments on Part 2 responses: 
Provide Estimates at 
Product Size Reporting CARD 


1. Number of anticipated Software Requirements, not including External Interface Requirements (unless noted in Data 
Dictionary) 


2. Number of anticipated External Interface Requirements (i.e., not under project control — 


Code Size Measures for items 4 through 6. For each, indicate S for physical SLOC (carriage returns); Snc for noncomment SLOC only; 
for logical statements; or provide abbreviation _-_—soaand explain in associated Data Dictionary. 


4. Expected amount of New Code to be developed and delivered (Sizein fF 
5. Expected amount of Modified Code to be developed and delivered (Size in ) fF 
6. Expected amount of Unmodified, Reused Code to be developed and delivered (Size in ) fF 
Comments on Part 3 responses: 


DD Form 2630-1 Page 1 of 2 
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Figure B-6: USA Software Resources Data Report — Initial Government Report (Page 1). 
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Software Resources Data Report: Initial Government Report - Sample 


Page 2: Project Resources, Schedule and Quality (Expected) 


4. Resource and Schedule Reporting Provide Estimates at CARD 


Counting from month 1 at contract award, provide Expected Start and End Month 
for each activity shown. Provide the Expected Total Labor Hours for each activity 
shown. 


Expected Expected Expected 
Start Month | End Month Total Hours 


The following seven items should contain estimates that account for all direct hours required for the 
software development project (use item 7 for any direct hours not accounted for in items 1 through 6). 
Explain any contribution of indirect hours in the associated Data Dictionary. 


Se 
ea 
Se 
fsmmcimnmevgmnemmimmem ff 
Se 
Fenmsieemmrowemmm YY 


7. All Other Direct Software Engineering Development Effort (Describe: 


) Estimate hours only: 


Comments on Part 4 responses: 


5. Product Quality Reporting (optional) 


One of the following items should be completed as a record of the expected reliability of the developed system. 


1a. Required Mean Time to Serious or Critical Defect (MTTD) at Delivery (provide specific definition in 
associated Data Dictionary): 


1b. Alternatively, use analogy to provide some measure that compares the required reliability of this system with the nominal 
reliability for systems of this type. Use the associated Data Dictionary to provide details about the analogous systems and 
any definitions of reliability used in this response. 


Comments on Part 5 responses: 
Filename and Revision Date of Applicable Software Resources Data Report Data Dictionary: 
aa - oo ed eal ve i aa 
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Figure B-7: Software Resources Data Report — Initial Government Report (Page 2). 
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Software Resources Data Report: Initial Developer Report - Sample 
Due 60 Days After Contract Award and 60 Days After Start of Any Release or Build 
Page 1: Report Context, Project Description and Size 


1. Report Context 


1. System/Element Name (version/release): 2. Report As Of: 


3. Authorizing Vehicle (MOU, contract/amendment, etc.): 4. Reporting Event: Project/Release Start 
Submission # 


(Supersedes # , if applicable) 


Description of Planned Development Organization 


5. Name of Development Organization: : 8. Lead Evaluator: 
or equivalent): 


7. Certification Date: 9. Affiliation: 


10. Precedents (list up to five similar systems by the same organization or team): 
Comments on Part 1 responses: 


Planned Development Process 
17. Primary Language (planned): 


19. Secondary Language (planned): 
21. List COTS/GOTS Applications Planned: 


22. Peak staff (maximum team size in FTE) expected to work on and charge to this project: 


23. Percent personnel expected to be: Highly experienced in domain: ___% _Nominally experienced: ___%_Entry level, no experience: __—(% 


Comments on Part 2 responses: 


; : Estimates at time of 
3. Product Size Reporting palate pat d 


1. Number of Software Requirements, not including External Interface Requirements (unless noted in associated Data 
Dictionary) expected to be satisfied by delivered software product 


2. Number of External Interface Requirements (i.e., not under project control) expected to be satisfied by delivered 
software product 


Code Size Measures for items 4 through 6. For each, indicate S for physical SLOC (carriage returns); Snc for noncomment 
SLOC only; LS for logical statements; or provide abbreviation and explain in associated Data Dictionary. 


4. Expected amount of New Code to be developed and delivered (Size in 
5. Expected amount of Modified Code to be developed and delivered (Size in 


) 
) 
6. Expected amount of Unmodified, Reused Code to be developed and delivered (Size in ) 


Comments on Part 3 responses: 


DD Form 2630-2 Page 1 of 2 
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Figure B-8: Software Resources Data Report — Initial Developer Report (Page 1). 
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Software Resources Data Report: Initial Developer Report - Sample 
Page 2: Project Resources and Schedule (Expected) 


Resource and Schedule Reporting Provide estimates at Contract Award 


Counting from month 1 at contract award, provide Expected Start and End Month 
for each activity shown. Provide the Expected Total Labor Hours for each activity 
shown. 


Expected Expected Expected 
Start Month | End Month Total Hours 


The following seven items should contain estimates that account for all direct hours required for the software 
development project (use item 7 for any direct hours not accounted for in items 1 through 6). Explain any 
contribution of indirect hours or uncompensated overtime in the associated Data Dictionary. 


a 
re 
ee 
fener | 


5. Software Qualification Testing Pf] 
6. Software Developmental Test and Evaluation Pf] 


7. All Other Direct Software Engineering Development Effort (Describe: 


) Estimate hours only: 


Comments on Part 4 responses: 


5. Product Quality Reporting (optional) 

i No Quality Reporting required at Contract Start. 

Filename and Revision Date of Applicable Software Resources Data Report Data Dictionary: 

— ne eiais de er 
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Figure B-9: Software resources Data Report — Initial Developer Report (Page 2). 
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Software Resources Data Report: Final Developer Report - Sample 
Page 1: Report Context, Project Description and Size 
4. Report Context 


1. System/Element Name (version/release): 2. Report As Of: 


3. Authorizing Vehicle (MOU, contract/amendment, etc.): 4. Reporting Event: Contract/Release End 
Submission # 


(Supersedes # , if applicable) 
Description of Actual Development Organization 
5. Development Organization: 


10. Precedents (list up to five similar systems by the same organization or team): 


Comments on Part 1 responses: 


21. List COTS/GOTS Applications Used: 


22. Peak staff (maximum team size in FTE) that worked on and charged to this project: 


23. Percent of personnel that was: Highly experienced in domain: __% Nominally experienced: __%_—_— Entry level, no experience: __—% 


Comments on Part 2 responses: 
F A Provide Actuals at 
3. Product Size Reporting Final Delivery 


1. Number of Software Requirements, not including External Interface Requirements (unless noted in associated Data 
Dictionary) 


2. Number of External Interface Requirements (i.e., not under project control) fo 
3. Amount of Requirements Volatility encountered during development (1=Very Low .. 5=Very High) pe 


Code Size Measures for items 4 through 6. For each, indicate S for physical SLOC (carriage returns); Sne for noncomment 
SLOC only; LS for logical statements; or provide abbreviation and explain in associated Data Dictionary. 


4. Amount of New Code developed and delivered (Size in ) 


5. Amount of Modified Code developed and delivered (Size in ) 


6. Amount of Unmodified, Reused Code developed and delivered (Size in 


Comments on Part 3 responses: 


DD Form 2630-3 Page 1 of 2 
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Figure B-10: Software Resources Data Report — Final Developer Report (Page 1). 
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Software Resources Data Report: Final Developer Report - Sample 
Page 2: Project Resources, Schedule and Quality 
ing 


4. Resource and Schedule Report Provide Actuals at Final Delivery 


Counting from month 1 at contract award, provide Actual Start and End Month for 
each activity shown. Provide the Actual Total Labor Hours for each activity Total Hours 
shown. 


The following seven items should account for all direct hours charged to the software development 
project (use item 7 for any direct hours not accounted for in items 1 through 6). Explain any contribution 
of indirect hours in the associated Data Dictionary. 


1. Software Requirements Analysis a ae |e 
2. Software Architecture and Detailed Design PE 
3. Software Coding and Unit Testing a 


4. Software Integration and System/Software Integration 


5. Software Qualification Testing 


2a. Measured or computed Mean Time to Serious or Critical Defect (MTTD) at Delivery. Provide the specific definition 
of this measure in the associated Data Dictionary. 


2b. Alternatively, use analogy to compare the observed or computed reliability of this system with the nominal reliability for similar systems. 
Use the associated Data Dictionary to provide details about the analogous systems and any definitions of reliability used in this response. 


Comments on Part 5 responses: 


Filename and Revision Date of Applicable Software Resources Data Report Data Dictionary: 


DD Form 2630-3 Page 2 of 2 
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Figure B-11: Software Resources Data Report — Final Developer Report (Page 2). 
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Annex C —- EXAMPLE OF A TYPICAL LCC QUESTIONNAIRE 


In order to conduct a comprehensive life cycle cost analysis, information from the supplier is required. 
This information can be gathered through formal approaches such as a Request for Information (RFD or a 
more contractual Request for Quotation (RFQ) or Request for Proposal (RFP). If there is more than one 
potential supplier a tender competition may be conducted and an Invitation to Tender (ITT) will be issued. 
All the requests or ITT will contain some sort of life cycle cost questionnaire. Based on the experience of 
the participating nations in SAS-054, this annex gives an overview of elements that should be requested in 
a life cycle cost questionnaire. 


There is no standard life cycle cost questionnaire that can be used for all acquisition projects. Any life 
cycle cost questionnaire should always be tailored for each specific case. 


The questionnaire should first request information and data for all the ground rules and assumptions that 
have been taken into account in developing the cost estimates, preferably in the form of an MDAL (Master 
Data and Assumptions List) or CARD (Cost Analysis Requirements Document). 
Other information should also be requested such as: 

¢ Economic uncertainties; 

¢ Demands for payment in advance; 

¢ Different financial solutions; and 

¢ Benefits with “dual use” (e.g. scale factors with more users). 
At the very minimum, the total cost should be broken down into the following main cost areas (data that 
must be provided is shown in italics). 

¢ Initial investment or acquisition costs, including development; 

e Annual cost of operation and maintenance; and 


¢ Cost of phasing out and disposing. 
These main cost areas should, if possible, be broken down into the following cost elements: 
1) Initial Investment or Acquisition Costs, including Development 


¢ Procurement costs of the system 
Number and price per system 


Number and price per subsystem 


¢ Initial logistic support costs 
Initial costs for training (operating and maintenance personnel) 
Initial costs for spare parts 
Initial costs of Test and Support Equipment 
Initial cost of documentation 
Initial costs computer hardware (e.g. interfaces) and software 
Initial costs for infrastructure (facilities) 


Initial costs consumables 
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2) 


3) 


Preparation costs 
Costs of testing prototypes 


Costs of installation of the system and adapting the system to the country specific 
circumstances 


Costs for testing and acceptance system 


Costs of transport and distribution of the system 


Other procurement costs 
e.g. costs of technical assistance (CETS or FSR) 


Annual Cost of Operation 


Costs of operating personnel 


These costs will be not be provided by the supplier. However, in order to compare the number 
and type of operating personnel available and required, the supplier shall provide the number and 
type of personnel required to operate the system. 


Costs of periodic training operating personnel (both training new personnel and refreshment 
courses). 


¢ Costs of update service of operating documentation. 
¢ Costs of use infrastructure (facilities) for operating the system. 


¢ Costs of use of consumables. 


The costs may not be provided by the supplier. However, these costs can be calculated using the 
unit price and the estimated use of fuel, oils, lubricants, ammunition and maintenance material. 


Annual Cost of Maintenance 


Costs of maintenance personnel at operational level. 


These costs will be not be provided by the supplier. However, in order to compare the number 
and type of maintenance personnel available and required, the supplier shall provide the number 
and type (education level and type) of maintenance personnel required at operational level. 


Costs of maintenance personnel at depot level. 


These costs can be calculated by using the number of Man Maintenance Hours (MMH) at depot 
level for each type of personnel spent on maintenance on the system. 


Costs of other personnel. 


Costs of any other personnel not yet included. 


Costs of periodic training maintenance personnel (both training new personnel and refreshment 
courses). 


Costs of spare parts consumption. 
Costs for outsourcing maintenance. 


Costs of use of Test and Support Equipment. 
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¢ Costs of update service of maintenance documentation. 

¢ Costs of updating computer hardware and software. 

* Costs for use of infrastructure required for maintenance. 
¢ Package, handling, storage and transportation costs. 


¢ Cost of recommended and mandatory modifications of the system. 


4) Cost of Phasing Out and Disposing 
¢ Possible retail price of the system or subsystems, after xx years of operation. 
¢ Costs to dismantle or destruct the system or subsystems. 
¢ Possible retail price of logistic support elements. 
* Costs to destruct logistic support elements. 


For production contracts the following information should be specified for each cost element of the CBS 
in the price quotation: 


e Labour cost — hours times hourly cost for each level of resources (e.g. Senior/Junior Engineer, 
Technician, etc.), including any external sub-contracted work; 


¢ Material cost — if necessary, specifying costs for materials procured in country, in Euro countries 
or outside Euro countries — including a 2% surplus for scrap and also customs and insurance 
expenses where applicable; and 


¢ Overheads — including material handling and other expenses (e.g. royalties, contract duties, 
production investment) where appropriate. 
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Annex D - LIFE CYCLE COSTING DEFINITIONS 


Affordability: Can be considered as the degree to which the life cycle cost of an acquisition programme is 
in consonance with the long-range investment and force structure plans of national defence 
administrations. Affordability procedures establish the basis for fostering greater programme stability 
through the assessment of programme affordability and the determination of affordability constraints. 
In this context: 


¢ In consonance means delivering systems that meet the customer’s needs and budget. 
¢ Programme stability means working towards sustainable opportunities. 


¢ Assessment means creating a programme management strategy that guarantees programme 
viability. 


¢ Determination of affordability constraints means bringing affordability to the foreground to avoid 
misconceptions in management and engineering which may ultimately lead to unaffordable design 
solutions. 


Analogy: A technique used to estimate a cost based on historical data for an analogous system or 
subsystem. In this method, a currently fielded system, similar in design and operation to the proposed 
system, is used as a basis for the analogy. The cost of the proposed system is then estimated by adjusting 
the historical cost of the current system to account for differences (between the proposed and current 
systems). Such adjustments can be made through the use of factors (sometimes called scaling parameters) 
that represent differences in size, performance, technology, and/or complexity. Adjustment factors based 
on quantitative data are usually preferable to adjustment factors based on judgments from subject-matter 
experts. 


Analysis of Alternatives (AoA): An analytical comparison of the operational effectiveness, suitability, 
and life-cycle cost of alternatives that satisfy established capability needs. Initially, the AoA process 
typically explores numerous conceptual solutions with the goal of identifying the most promising options, 
thereby guiding the concept refinement phase. Subsequently, the AoA is used to justify the rationale for 
formal initiation of the acquisition program. An AoA normally is not required for production decisions 
unless significant changes to threats, costs, or technology have occurred. 


Cost Analysis Requirements Description: The CARD is used to formally describe Acquisition Category 
I and JA programs and systems in the United States DoD for purposes of preparing both the program 
office cost estimate (and the Component cost position, if applicable) and the OSD CAIG independent cost 
estimate. The CARD is provided in support of major milestone decision points (Milestone B, Milestone C, 
or the full-rate production decision review). In addition, for major AIS programs, the CARD is prepared 
whenever an economic analysis is required. The CARD is prepared by the program office and approved by 
the DoD Component Program Executive Officer (PEO). For joint programs, the CARD includes the 
common programme agreed to by all participating DoD Components as well as all unique program 
requirements of the participating DoD Components. DoD 5000.4-M, DoD Cost Analysis Guidance and 
Procedures, Chapter 1, provides further guidelines for the preparation of the CARD. 


Note that the CARD, ideally, provides both narratives and tabular data in sufficient detail for the 
programme to be costed. Naturally, the exact level of detail provided in the CARD will depend on the 
maturity of the programme. Programmes at Milestone B are less well-defined than programmes at 
Milestone C or at full-rate production. In cases where there are gaps or uncertainties in the various 
programme descriptions, these uncertainties are, again ideally, acknowledged as such in the CARD. 
Dealing with program uncertainty in the CARD greatly facilitates subsequent sensitivity or 
quantitative risk analyses in the life-cycle cost estimate. 
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Contractor Cost Data Reporting System: The primary means within the U.S. Department of Defense to 
systematically collect data on the development and production costs incurred by contractors in performing 
DoD acquisition program contracts. CCDR reports can provide for each contract a display of incurred 
costs to date and estimated incurred costs at completion by elements of the work breakdown structure, 
with nonrecurring costs and recurring costs separately identified. In addition, CCDR reports can display 
incurred costs to date and estimated incurred costs at completion by functional category (manufacturing, 
engineering, etc.). Each functional category is broken out by direct labour hours and major cost element 
(direct labour, direct material, and overhead). The CCDR manual (which provides report formats and 
definitions, specific report examples, and other related information) can be found at the Defense Cost and 
Resource Center (DCARC) web site. The DCARC is the OSD office responsible for administering the 
CCDR system. 


Cost Element Structure (CES): A unit of costs to perform a task or to acquire an item. The cost 
estimated may be a single value or a range of values. 


Cost Estimating Relationships (CERs): Equations that relate the cost of a weapon system or subsystem 
(or some other dependent variable) to one or more technical, physical, or performance characteristics of 
that system. CERs can range from simple rules of thumb derived as the average of a couple of data points 
to multi-variate regressions complete with a host of related statistics. 


Cost Estimating (defined by SCEA, found in FAA Life Cycle Cost Estimating Handbook Sub-section 
2.2): The art of approximating the probable cost or value of something based on information available at 
the time. 


Cost model (taken from SAS-054 POW): A cost model is a set of mathematical and/or statistical 
relationships arranged in a systematic sequence to formulate a cost methodology in which outputs, namely 
cost estimates, are derived from inputs. These inputs comprise a series of equations, ground rules, 
assumptions, relationships, constants, and variables, which describe and define the situation or condition 
being studied. Cost models can vary from a simple one- formula model to an extremely complex model 
that involves hundreds or even thousands of calculations. A cost model is therefore an abstraction of 
reality, which can be the whole or part of a life cycle cost. 


Data (taken from Joint publication 1-02 DoD Dictionary of Military and Associated Terms, 12/04/01, 
amend. 05/06/03 http://centre.chots.mod.uk/jel/pdfdocs/jel/diction/termdict.pdf): Representation of facts, 
concepts, or instructions in a formalised manner suitable for communication, interpretation, or processing 
by humans or by automatic means. Any representations such as characters or analog quantities to which 
meaning is, or might, be assigned. 


Earned Value Management (EVM): A project management technique that objectively tracks physical 
accomplishment of work. EVM has the unique ability to combine measurements of technical performance 
(i.e. accomplishment of planned work), schedule performance (i.e. behind/ahead of schedule), and cost 
performance (i.e. under/over budget) within a single integrated methodology. EVM provides an early 
warning of performance problems while there is time for corrective action. In addition, EVM improves the 
definition of project scope, prevents scope creep, communicates objective progress to stakeholders, and 
keeps the project team focused on achieving progress. 


EVM emerged as a financial analysis specialty in United States Government programmes in the 
1960s, but it has since become a significant branch of project management. Implementations of EVM 
can be scaled to fit projects of all sizes and complexity. 


Economic Analysis: A systematic approach to the problem of choosing the best method of allocating scarce 
resources to achieve a given objective. A sound economic analysis recognises that there are alternative ways 
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to meet a given objective and that each alternative requires certain resources and produces certain results. 
To achieve a systematic evaluation, the economic analysis process employs the following two principles: 


Each feasible alternative for meeting an objective must be considered, and its life-cycle costs and 
benefits evaluated. All the costs and benefits are adjusted to “present value” by using discount factors 
to account for the time value of money. Both the size and timing of costs and benefits are important. 


Engineering Estimates: With this technique, the system being costed is broken down into lower-level 
components (such as parts or assemblies), each of which is estimated separately for direct labour, direct 
material, and other costs. Engineering estimates for direct labour hours may be based on analyses of 
engineering drawings and contractor or industry-wide standards. Engineering estimates for direct material 
may be based on discrete raw material and purchase part requirements. The remaining elements of cost 
(such as quality control or various overhead charges) may be factored from the direct labour and material 
costs. The various discrete cost estimates are aggregated by simple algebraic equations (hence the 
common name “bottoms-up” estimate). The use of engineering estimates requires extensive knowledge of 
a system’s (and its components’) characteristics, and lots of detailed data. 


Estimation Methods: Popular methods of estimating life cycle costs include analogy, engineering 
(bottoms-up), and parametric. 


Inflation: A rise in the general level of prices, as measured against some baseline of purchasing power. 
Ina U.S. DoD cost estimating environment, the following terms regarding inflation are commonly used: 
Base Year: A point of reference that represents a fixed price level. In a weapon system acquisition 
context, this is often the fiscal year when a programme was initially funded. 
Budget 


Escalation: Application of the effects of inflation to a dollar amount. De-escalation is the reverse, 
or removing the effects of inflation from a dollar value. 


Expenditures: A charge against available funds or the actual payment of funds evidenced by 
voucher, claim, or other document approved by competent authority. 


Outlay Profiles: Indicates the rate at which dollars in each appropriation are expected to be 
expended based on historical experience. 


Total Obligational Authority (TOA): The yearly value the defense programme, roughly equal to 
new budget authority plus any prior year balance still available for obligation. 


Dollars 


Base-Year Dollars: The money or prices expressed in terms of the purchasing power prevailing 
in a specified base year. 


Constant-Year Dollars: Dollars expressed in their value at the time of any specified year, which 
may, but does not have to be, the base year. Also called “constant dollars.” 


Current-Year Dollars: Money or prices expressed in terms of values actually observed in the 
economy at any given time. That is, 2010 Current Year dollars would be the actual amount of 
dollars you’d have to pay in 2010 to purchase something of a given value in that year. 


Then-Year Dollars: Constant or base-year dollars deflated or inflated through the use of indices 
to show total money needed to buy those goods and services at the time expenditures are actually 
made. Or, put another way, Total Obligational Authority (TOA) that includes a slice of inflation to 
cover escalation of expenditures over a multi-year period. 
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Indices 


Raw: An annual compounding of escalation or inflation rates from a fixed point of reference, 
designated 100 or 1.0, and called the base year. 


Weighted: A combination of raw escalation or inflation indices and outlay rates that indicates the 
amount of escalation occurring over the entire period of time that is required to expend TOA. 


Learning and Experience Curves: The learning curve effect and the closely related experience curve 
effect express the relationship between experience and efficiency. As individuals and/or organizations get 
more experienced at a task, they usually become more efficient at them. Both concepts originate in the 
adage, “practice makes perfect,” and both concepts are opposite to the popular misnomer that a “steep” 
learning curve means that something is hard to learn. In fact, a “steep” learning curve implies that 
something gets easier quickly. 


The learning curve effect states that the more times a task has been performed, the less time will be 
required on each subsequent iteration. It was discovered by the 19th-century German psychologist 
Hermann Ebbinghaus. This relationship was probably first quantified in 1936 at Wright-Patterson Air 
Force Base in the United States, where it was determined that every time total aircraft production 
doubled, the required labour time decreased by 10 to 15 percent. 


The experience curve effect is broader in scope than the learning curve effect encompassing far more 
than just labour time. It states that the more often a task is performed the lower will be the cost of 
doing it. The task can be the production of any good or service. Each time cumulative volume doubles, 
value added costs (including administration, marketing, distribution, and manufacturing) fall by a 
constant and predictable percentage. 


Life Cycle (taken from Joint publication 1-02 DoD Dictionary of Military and Associated Terms, 12/04/01, 
amend. 05/06/03 http://centre.chots.mod.uk/jel/pdfdocs/jel/diction/termdict.pdf): The total phases through 
which an item passes from the time it is initially developed until the time it is either consumed in use or 
disposed of as being excess to all known materiel requirements. 


Life Cycle Cost (LCC) (taken from RTO-TR-058 report): LCC consists of all direct costs plus indirect- 
variable costs associated with the procurement, O&S and disposal of the system. Indirect costs may 
include linked costs such as additional common support equipment, additional administrative personnel 
and non-linked costs such as new recruiters to recruit additional personnel. All indirect costs related to 
activities or resources that are not affected by the introduction of the system are not part of LCC. 


Nature of Decision: LCC comprises the marginal costs (both direct and indirect) of introducing a new 
equipment or capability. LCC is used as a minimum for the analysis of alternatives, it does not include 
notional allocation of costs, whereas TOC and WLC might do so. LCC is used to compare options of 
alternatives, and often for economic analyses. 


Military Requirements: (taken from Joint publication 1-02 DoD Dictionary of Military and Associated 
Terms, 12/04/01, amend. 05/06/03 _ http://centre.chots.mod.uk/jel/pdfdocs/jel/diction/termdict.pdf): 
An established need justifying the timely allocation of resources to achieve a capability to accomplish 
approved military objectives, missions, or tasks. Also called operational requirement. 


Operations: (taken from JWP 0-01.1, Edition 6, http://www.chots.mod.uk/jointwar/): A military action or 
the carrying out of a strategic, tactical service, training, or administrative military mission: the process of 


carrying on combat, including movement, supply, attack, defence and manoeuvre needed to gain the 
objectives of any battle or campaign. 


Parametric Estimating: This technique uses regression or other statistical methods to develop Cost 
Estimating Relationships (CERs). A CER is an equation used to estimate a given cost element using an 
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established relationship with one or more independent variables. The relationship may be mathematically 
simple (e.g. a ratio) or it may involve a complex equation (often derived from regression analysis of 
historical systems or subsystems). CERs should be current, applicable to the system or subsystem in 
question, and appropriate for the range of data being considered. 


Product (taken from http://www.ams.mod.uk): The results of activities or processes. It includes (taken from 
RTO-TR-058 report) the products delivered to the user and the specific means required for developing and 
manufacturing these products. The products delivered to the user encompass the main system (aircraft, tank, 
etc.) and its support elements (spares, support equipment, facilities, documentation, etc.). 


Project (taken from http://www.ams.mod.uk): The structure of authorities, resources and capabilities that 
supplies defined products and / or services within agreed time, cost and performance criteria. 


Risk is exposure to loss. Or, in a weapon-system acquisition context, it is “a measure of the potential 
inability to achieve overall program objectives within defined cost, schedule, and technical constraints and 
has two components: (1) the probability/ikelihood of failing to achieve a particular outcome, and (2) the 
consequences/impacts of failing to achieve that outcome”. 


Standardised Product: (taken from JWP 0-01.1, Edition 6, http://www.chots.mod.uk/jointwar/): 
A product that conforms to specifications resulting from the same or equivalent technical requirement. 
NATO standardised products are identified by a NATO code number. 


Standardised Requirement (taken from JWP 0-01.1, Edition 6, http://www.chots.mod.uk/jointwar/): 
Within NATO, a broad statement identifying the levels of standardisation that should be achieved within 
specific areas of operations, materiel, administration and the related procedures”. 


Sensitivity Analysis: Attempts to demonstrate how the cost estimate would change if one or more 
assumptions change. Typically, for the high-cost elements, the analyst identifies the relevant cost-drivers, 
and then examines how costs vary with changes in the cost-driver values. For example, a sensitivity 
analysis might examine how maintenance manning varies with different assumptions about system 
reliability and maintainability values, or how system manufacturing labour and material costs vary with 
system weight growth. In good sensitivity analyses, the cost-drivers are not changed by arbitrary 
plus/minus percentages, but rather by a careful assessment of the underlying risks. Sensitivity analysis is 
useful for identifying critical estimating assumptions, but has limited utility in providing a comprehensive 
sense of overall uncertainty. 


System (taken from http://www.ams.mod.uk): A human-made entity with a distinguishing and defined 
purpose that draws on integrated, constituent parts, each of which does not individually possess the 
required overall characteristics or purpose. 


Task (taken from RTO-TR-058 report): Is the most elementary process or piece of work to be done, 
especially one done regularly to obtain an expected result and specified in terms of performance, cost and 
time. The performance of a task is entrusted to an identified actor and usually requires human, material 
and financial resources allocation. 


Total Ownership Cost (TOC): Total ownership cost consists of the elements of a programme’s life cycle 
cost, as well as other infrastructure or business processes costs not necessarily attributable to the 
programme. This may include items such as common support equipment, common facilities, personnel 
required for unit command, administration, supervision, operations planning and control, fuel and 
munitions handling. 


Nature of Decision: TOC represents all costs associated with the ownership of a system except non- 
linked fixed costs that are related to the running of the organisation. TOC is used for budgeting 
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purposes, determining the use of services between systems, for optimisation purposes and for financial 
analysis. 


Uncertainty: Is the indefiniteness or variance of an event. It captures the phenomenon of observations, 
either favourable or unfavourable, falling to the left and right of a mean or median value. 


VAMOSC: The U.S.’s Visibility and Management of Operating and Support Costs (VAMOSC) system is 
the set of data and data management systems for the collection, display and cataloguing of historical O&S 
costs, related data, and associated factors that determine those costs, by individual defence programme. 
Each military department in the U.S. DoD is responsible for developing their own VAMOSC systems; 
hence, there is no single VAMOSC system, but rather several closely related but independent VAMOSC 
systems. VAMOSC data can be displayed in several different formats, including the CAIG standard cost 
element structure. Data can be obtained for entire systems, or at lower levels of detail. VAMOSC provides 
not only cost data, but related non-cost data (such as OPTEMPO or maintenance man-hours) as well. 
This type of data is useful for analogy estimates (between proposed systems and appropriate predecessor 
or reference systems) and for “bottoms-up” engineering estimates (for fielded systems or components, 
possibly adjusted for projected reliability and maintainability growth). VAMOSC data should always be 
carefully examined before use in a cost estimate. The data should be displayed over a period of a few 
years (not just a single year), and stratified by different sources (such as major command or base). 
This should be done so that abnormal outliers in the data can be identified, investigated, and resolved as 
necessary. 


Whole Life Cost (WLC) (taken from RTO-TR-058 report): WLC consists of all elements that are part of 
TOC plus indirect, fixed, non-linked costs. These latter may include items such as family housing, medical 
services, ceremonial units, basic training, headquarters and staff, academies, recruiters. In WLC all costs 
or expenses that are made by the organisation are attributed to the systems or products they produce. 


Nature of Decision: As WLC represents the total budget provision including such element as 
headquarters costs, it allows the visibility of the complete allocation of funds. WLC is used for a 
strategic view and high level studies. 


Work Breakdown Structure (WBS): A technique for representing all the components, software, services 
and data contained in the project scope statement. It establishes a hierarchical structure or product oriented 
“family tree” of elements. It is used to organise, define and graphically display all the work items or work 
packages to be done to accomplish the project’s objectives. 
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The following descriptions of the PAPS milestones have been drawn directly from AAP-20. See Chapter 
3.2 for a description of the PAPS life cycle phases. Figure 3.1 gives an overview of the milestones and 
their relationship with the PAPS life cycle phases. 


E.1 MISSION NEED DOCUMENT (MND) 


A statement based on a mission analysis, identifying in broad outline a quantitative or qualitative 
operational deficiency that cannot be solved satisfactorily with existing or planned forces and/or 
equipment. 


E.2 OUTLINE NATO STAFF TARGET (ONST) 


A very broad outline of the function and desired performance of a new weapon or equipment, to satisfy a 
mission need, before the possibilities of achievement and the financial aspects have been examined. 
Contains operational characteristics, details of the threat, desired capability and a general indication of 
scope in particular and broad cost parameters whenever possible. Sufficient detail is given to enable a 
pre-feasibility study (or studies) to be carried out. 


E.3. NATO STAFF TARGET (NST) 


A broad outline of the function and desired performance of new equipment or weapon system(s), before 
the feasibility or method of meeting the requirement, or other implications have been fully assessed. Based 
upon the findings of any pre-feasibility study (studies) the NATO Staff Target lists, in greater detail, 
operational characteristics and certain technical specifications which are desired and which have been 
shown to be broadly feasible. It may also contain cost parameters when required. The NST is used as a 
basis for the Request for Proposals (RFP) from industry for a feasibility study of candidate system 
solutions. 


E.4 NATO STAFF REQUIREMENT (NSR) 


A detailed statement of the required design parameters and operational performance of the equipment or 
weapon system(s). This document represents the specification of the system upon which project definition 
is based. 


E.5 NATO DESIGN AND DEVELOPMENT OBJECTIVE (NADDO) 


An outline statement which covers the evaluation of design proposals in relation to the user requirement, 
the statement of agreed characteristics, and the design and technical requirement specifications. It includes 
as far as possible demonstrated achievement of stated requirements and objectives for the future, aimed at 
ensuring full system integration. 


E.6 NATO PRODUCTION OBJECTIVE (NAPO) 


An outline statement of the manufacturing processes, manpower and facilities required for production of 
the equipment, including an outline production programme based on cost plans, quality control 
requirements, and the stated production specification. 
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E.7 NATO IN-SERVICE GOALS (NISEG) 


A statement of general utilisation intentions for equipment including reference to national or co-operative 
logistics and training arrangements. 


E.8. NATIONAL DISENGAGEMENT INTENTION (NADI) 


A statement of a decision to withdraw equipment from operational utilisation including forecast dates, 
quantities, and other relevant information such as age, condition of equipment, and availability of spare 
parts. 
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